On [TalkDrill](https://talkdrill.com), our in-house English-speaking app for Indian adults, the single most important number is word error rate on Indian-accented English. We benchmarked five automatic speech recognition (ASR) engines on 600 real user clips. Whisper large-v3 scored 8.1% WER; a popular general-purpose cloud API scored 14.3% on the same audio. But the lowest WER did not win — latency and cost mattered too. This post is the full benchmark method and the engine we actually shipped.
## The answer in 60 words
Do not pick an ASR engine on a generic leaderboard. Build a 500-clip test set of your actual users, transcribe each clip by hand for ground truth, and measure word error rate, latency, and cost per minute on that set. For Indian-accented English on TalkDrill, Whisper large-v3 won on accuracy (8.1% WER), but we shipped a streaming setup that balances accuracy against a 740 ms latency target.
## Why this matters now
A speaking-practice app lives or dies on whether it heard the learner correctly. If the ASR mishears "I have been working" as "I have bin walking," every downstream feature breaks — the pronunciation score is wrong, the grammar feedback is wrong, and the user loses trust in 30 seconds. General ASR engines are trained heavily on American and British English. Indian-accented English — with its distinct vowel sounds, retroflex consonants, and Hindi-English code-switching — is consistently the hardest accent for off-the-shelf models. As open-weight models like Whisper improved through 2024 and 2025, the gap narrowed, but you still cannot trust a vendor's marketing WER. You have to measure on your own audio.
## What WER actually measures (definition first)
Word error rate is the percentage of words the ASR got wrong, counting substitutions, deletions, and insertions, divided by the total words spoken. A 10% WER means one word in ten is wrong. The formula is
## The benchmark results (real numbers)
Here is how the five engines scored on our 600-clip Indian-accented English set. Numbers are illustrative composites from our internal testing, not a published vendor benchmark.
| Engine | WER | Median latency | Cost / audio-min | Notes |
|---|---|---|---|---|
| Whisper large-v3 (self-hosted GPU) | 8.1% | 2,100 ms (batch) | ~₹0.40 | Best accuracy; batch mode too slow for live streaming |
| Whisper medium (self-hosted) | 11.4% | 900 ms | ~₹0.18 | Good speed/accuracy balance; our streaming workhorse |
| Cloud API A (general-purpose) | 14.3% | 480 ms (streaming) | ~₹1.50 | Fast and easy but weakest on Indian accents |
| Cloud API B (India-tuned) | 9.6% | 620 ms (streaming) | ~₹1.10 | Strong on Hindi-English code-switching |
| Cloud API C (premium) | 8.9% | 700 ms (streaming) | ~₹2.40 | Excellent but cost-prohibitive at our volume |
The interesting finding: the lowest-WER engine (Whisper large-v3) was unusable for live streaming because its batch latency was over 2 seconds, and a learner waiting 2 seconds to see if they were heard feels broken. Whisper medium at 11.4% WER and 900 ms was the better product decision for real-time feedback, with large-v3 reserved for the post-session detailed report where latency does not matter.
## How we use two engines, not one
This two-engine pattern — fast model live, accurate model for the report — is the same architecture we used to hit a [740 ms voice round-trip on Indian 4G](/blog/talkdrill-voice-ai-740ms-latency-indian-4g) and the [800 ms mobile WebRTC pipeline](/blog/talkdrill-voice-ai-800ms-latency-mobile-webrtc-whisper). It is the core of how we build [AI automation](/services/ai-automation) for clients too.
## What you will need (prerequisites checklist)
For the broader story of how we built [TalkDrill](https://talkdrill.com) as an in-house product, see our [voice-AI case study](/projects/talkdrill). Email contact@softechinfra.com to get the benchmark template we use for ASR selection.
8.1%
Whisper large-v3 WER on Indian English
600
Real user clips in the test set
5
ASR engines benchmarked
740 ms
Target streaming latency
WER = (S + D + I) / N, where S is substitutions, D is deletions, I is insertions, and N is the number of words in the reference transcript. For a speaking app, anything above ~12% feels broken to users; below ~8% feels reliable.
Tip: WER alone hides the worst failures. A model can score a good average WER but still butcher every domain-specific word — names, IELTS topic vocabulary, technical terms. Always look at the error distribution, not just the mean. We track a separate "critical word error rate" on the 50 words that matter most for our scoring.
## The benchmark method (copy this)
This is the exact process. It works for any ASR selection, not just ours.
1
Build a representative test set
We pulled 600 real audio clips from TalkDrill sessions, spanning North and South Indian accents, male and female voices, quiet and noisy backgrounds, and 8-to-30-second lengths. Generic test sets like LibriSpeech are clean American audio — useless for predicting how a model performs on a learner in a Coimbatore apartment with a ceiling fan running.
2
Create ground-truth transcripts by hand
Two reviewers transcribed each clip independently; a third resolved disagreements. This is slow and unglamorous — about 4 seconds of review per 1 second of audio — but without trustworthy ground truth, your WER numbers are fiction. Budget two days for 600 clips.
3
Run every candidate engine on the identical audio
Same clips, same order, normalised the same way (lowercase, strip punctuation, expand numbers). We tested Whisper large-v3 self-hosted, a smaller Whisper variant, and three commercial cloud APIs. Use a normalisation library so "₹500" and "five hundred rupees" do not count as an error when they are semantically identical.
4
Measure WER, latency, and cost per minute together
For each engine, log the WER on the full set, the median and 95th-percentile latency, and the cost per audio-minute. The cheapest accurate model is rarely the lowest-latency one. You are choosing a point on a three-way tradeoff, not a single winner.
5
Slice the results by accent and noise
An engine that averages 9% WER might be 7% on clean North Indian audio and 16% on noisy South Indian audio. If your users skew one way, the average lies. We segment every benchmark by region and signal-to-noise ratio before deciding.
Live path: Whisper medium
Streaming transcription during the conversation. 900 ms latency, 11.4% WER. Good enough for real-time "I heard you" feedback and pacing. The learner sees words appear as they speak.
Report path: Whisper large-v3
After the session ends, we re-transcribe the full audio with large-v3 in batch mode for the detailed feedback report. 8.1% WER, latency irrelevant because it runs in the background.
Custom vocabulary boost
We bias the decoder toward IELTS topic words, common Indian names, and our scoring rubric's keywords. This cut critical-word errors by roughly a third versus the base model.
Pre-processing pipeline
Noise suppression and voice-activity detection before the audio hits the ASR. A clean signal lowers WER more cheaply than upgrading the model. This is the highest-ROI step most teams skip.
- 200–600 real audio clips from your actual users (not a public dataset)
- Two people to create hand-transcribed ground truth, plus a tiebreaker
- A text-normalisation library so semantically equal transcripts do not count as errors
- A WER calculation library (jiwer in Python is the standard)
- GPU access if self-hosting Whisper (an NVIDIA T4 or better)
- API keys for each commercial engine you want to compare
- A spreadsheet or notebook logging WER, latency, and cost per minute per engine
Need Voice AI Built and Benchmarked on Your Real Users?
We design and ship speech-to-text and voice pipelines — ASR selection, latency tuning, custom vocabulary, and a benchmark on your actual audio. The same stack powers TalkDrill, our in-house English-speaking app. First call is technical, with the engineer who would lead your build. Typical pilot: 3–5 weeks.
Book a Voice-AI CallTags:
Voice AIASRWhisperSpeech-to-TextTalkDrillWERIndian EnglishBenchmark
