Whisper LoRA Cuts Nepali Financial Speech WER 67.2%

Nepal's mobile payment apps are graphical walls for visually impaired users — every send, load, and balance command requires tapping through a screen. SpeakPay, a new single-author study from Biraj Subedi, attacks the problem with a voice-first wallet and a disciplined domain-adaptation experiment: fine-tuning Whisper large-v2 with LoRA on just 403 Nepali financial utterances cuts word error rate by 67.2% (from 129.95% zero-shot to 42.58%) and lifts transaction success roughly 20x. The data, code, and weights are all public.

Why Is Nepali Financial Speech Recognition Hard?

Whisper's zero-shot baseline fails catastrophically here — a 129.95% WER means it produces more errors than there are words, and it recognizes zero Devanagari numerals correctly. Three forces stack up: Nepali is a low-resource language with thin audio coverage in Whisper's training mix; financial commands are full of numerals (237 unique ones in this dataset — amounts, phone numbers, account digits), where a single misheard digit destroys a transaction; and the domain vocabulary (load, balance, send) is absent from generic speech data. The graphical-first design of Nepali fintech apps compounds it — there is no voice fallback at all.

How Does SpeakPay Adapt Whisper with LoRA?

The study contributes NepFinSpeech-403, a 403-utterance dataset of Nepali financial voice commands spanning send, load, and balance operations. Rather than full fine-tuning, the paper freezes Whisper large-v2 and trains LoRA adapters — a parameter-efficient route that keeps the base model intact and the adapter small enough to ship. The evaluation is task-grounded: beyond standard WER, they measure Devanagari numeral accuracy and a Transaction Success Rate that simulates whether a command would actually execute correctly end-to-end.

What Accuracy Does Whisper LoRA Achieve on Nepali Financial Speech?

On the held-out test set, LoRA adaptation drives WER from 129.95% to 42.58% — a 67.2% relative reduction — and Devanagari numeral recognition jumps from 0.0% to 73.9%. The practical metric is the headline: Transaction Success Rate rises from 1.67% to 33.33%, a ~20x gain that word-level metrics actually understate. The improvement is statistically solid (sign test, p < 10⁻¹⁷) and consistent across all three command types. The data-efficiency curve is the sleeper finding: 100 utterances suffice to halve zero-shot WER, with gains plateauing around 300 examples — a number that makes this recipe attractive for any other low-resource financial domain.

What Are the Limitations?

33.33% transaction success still means two of three voice commands fail. Remaining errors concentrate in numerals — the authors isolate systematic zero insertion/deletion and prefix hallucination as the dominant failure modes, which is exactly where money is lost. The dataset is a single speaker-style domain (403 utterances, one app's command grammar), so generalization across speakers, dialects, and noisy street audio is untested. LoRA kept the base model intact, but the adapter still inherits Whisper's weak Nepali priors — the ceiling may be the base model, not the method.

Why Should You Care If You're Building Voice Products?

This is a transferable template for a class of problems that is everywhere and underserved: transaction-critical speech in low-resource languages. The protocol — a small task-grounded dataset, LoRA on a frozen encoder-decoder, and evaluation on execution success rather than raw WER — costs almost nothing to replicate and is fully open. The 100-utterance tipping point is the number to remember: you may already have enough data sitting in logs to halve your error rate. If you build payment, health, or government voice interfaces outside the big languages, this paper is the closest thing to a field manual.