The first version of TalkDrill's pronunciation scorer told a fluent Hyderabad engineer his English was "poor." It was not. He pronounced "wine" and "vine" the same way — a textbook Indian-English feature, not an error. A scorer trained on American English flags that as a mistake every single time. TalkDrill, our in-house English-fluency app for Indian adults, now distinguishes a genuine pronunciation error from an expected L1 transfer feature. This post is how we did it, including the phoneme substitution table we built for seven Indian languages.
## The Answer in 60 Words
We score pronunciation at the phoneme level, then run the result through an L1-aware filter. If a speaker substitutes a sound in a way that is a known, systematic feature of their mother tongue — like the /v/-/w/ merger in Hindi-English — we do not penalise it. We only flag substitutions that block intelligibility. The accent stays; the genuine errors get coached.
## Why This Matters Now
There are hundreds of millions of L2 English speakers in India, and [research on Indian native-language phonemic influence on English](https://arxiv.org/pdf/2212.09284) is clear that speakers systematically carry L1 sounds into English — and that these patterns differ by mother tongue. A scorer built on American or British English treats every one of these as a defect. That is not feedback; it is telling a Tamil speaker their Tamil-ness is wrong. For an Indian product, getting this right is the whole job. The goal of TalkDrill is intelligible, confident English — not a fake American accent.
## The Core Distinction: Error vs. L1 Transfer Feature
Not every "non-standard" sound is a mistake. We split substitutions into three buckets and treat each differently.
## The Phoneme Substitution Table (7 Indian Languages)
This is the heart of the system. For each L1, we encode the systematic English substitutions documented in phonetics literature and confirmed against our own rated clips. When a speaker's detected substitution matches a row tagged "accepted," the scorer suppresses the penalty.
## The Calibration Loop (Where Trust Comes From)
A scorer is only as good as the humans it agrees with. We hand-rated 3,400 clips across the seven L1 groups, each scored by trained Indian-English raters for intelligibility, not nativeness. Then we tuned the filter until the machine's verdicts matched the human consensus. We hit 0.81 agreement (a strong inter-rater-style score) before shipping. This mirrors the [self-consistency and selective-human-review approach](https://www.mdpi.com/2504-4990/8/3/74) that improves grading reliability while keeping humans in the loop on the hard cases.
## When Not to Suppress
There is a real failure mode in the other direction: suppressing too much. If we accept every substitution as "just an accent," we stop teaching, and the user's intelligibility never improves. The line we hold is intelligibility. A retroflex /t/ is fine forever. But if consonant-cluster simplification makes "task" and "tax" indistinguishable to a listener, that is worth coaching, even though it is L1-driven. Accent-acceptance is not the same as never correcting anything. We tuned the table so the accepted bucket only contains features that do not cost comprehension.
7
L1 Languages Profiled
23
Substitution Rules in the Table
3,400
Human-Rated Clips for Calibration
0.81
Agreement With Human Raters
Accepted L1 Feature
A systematic mother-tongue transfer that does not block understanding — retroflex /t/ /d/, the /v/-/w/ merger, dental /θ/. We never mark these down. They are accent, not error.
Intelligibility-Blocking
A substitution that causes a real listener to mishear the word — dropping a consonant cluster so "school" becomes "skool/iskool" to the point of confusion, or stress on the wrong syllable changing the word. We coach these.
Genuine Mispronunciation
Not an L1 pattern at all — a one-off wrong sound the same speaker gets right elsewhere. This is the real learning target, and the only thing we score hard.
Progress, Not Perfection
We score against intelligibility and the speaker's own baseline, not against a native reference. Improvement is measured as fewer blocking errors over time.
| L1 (Mother Tongue) | English Target Sound | Common Substitution | Example | Verdict |
|---|---|---|---|---|
| Hindi | /v/ vs /w/ | Merged to a single /ʋ/ | "wine" ≈ "vine" | Accepted feature |
| Hindi / most North Indian | /θ/ (think) | Dental /t̪/ | "thin" → "t̪hin" | Accepted feature |
| Hindi / most North Indian | /ð/ (this) | Dental /d̪/ | "that" → "d̪at" | Accepted feature |
| Most Indian L1s | /t/ /d/ (alveolar) | Retroflex /ʈ/ /ɖ/ | "dog", "time" | Accepted feature |
| Bengali | /s/ vs /ʃ/ | /s/ realised as /ʃ/ | "sip" → "ship"-like | Accepted if intelligible |
| Bengali | /v/ | /bʱ/ or /b/ | "very" → "bery" | Accepted feature |
| Tamil | Voiced stops /b/ /g/ initially | Devoiced to /p/ /k/ | "goal" → "kol"-like | Accepted if intelligible |
| Tamil / Telugu | /z/ | /s/ or /dʒ/ | "zoo" → "soo" | Accepted feature |
| Telugu | Initial consonant clusters | Vowel epenthesis | "school" → "ischool" | Coach if it blocks meaning |
| Marathi | /æ/ (cat) | /ɛ/ or /e/ | "bat" → "bet"-like | Coach (changes the word) |
| Gujarati | /f/ | Sometimes /pʰ/ | "fan" → "phan" | Accepted feature |
| Punjabi | /dʒ/ vs /z/ | /z/ → /dʒ/ | "zero" → "jero" | Accepted feature |
Why the verdict column matters: the same substitution can be fine in one word and confusing in another. Our rule is intelligibility-first: if a real listener would still hear the right word, it is an accepted accent feature. The "bat → bet" Marathi vowel shift gets coached because it produces a different valid English word.
## How the Scorer Actually Works
1
Forced alignment to expected phonemes
The user reads a known sentence, so we know the target phoneme sequence. We align their audio to it and get a per-phoneme acoustic score — how close each sound was to the expected one.
2
Detect the actual substitution
Where a phoneme scores low, we identify what the speaker said instead. Now we have a substitution pair: expected /v/, produced /w/.
3
Look up the L1-aware filter
We know the user's mother tongue from onboarding. We check the substitution against that L1's row in the table. If it is an accepted feature, the penalty is suppressed. If it is intelligibility-blocking, it is flagged for coaching.
4
Score against intelligibility, not nativeness
The final score reflects how understandable the speech is, plus progress against the speaker's own baseline. A heavy but clear accent scores well. Genuine blocking errors pull it down and generate a specific drill.
- Raters are themselves Indian-English speakers across multiple L1s — they will not flag a retroflex /t/ as wrong.
- Every clip rated by at least two humans; disagreements escalate to a third.
- The machine's accepted-feature suppressions are audited against rater notes monthly.
- New L1 patterns found in the wild get added to the table, then re-calibrated.
- We track false-negative coaching (telling someone an accent feature is an error) as the metric we most want at zero.
The mistake we made first: our v1 had no L1 awareness at all — it scored against a single American reference. It systematically punished South Indian speakers hardest, because their L1 features diverge most from that reference. We were encoding a bias as a "score." The L1 filter exists to remove exactly that bias.
## Real Example: The Hyderabad Engineer
The user who triggered this whole rebuild was a working engineer in Hyderabad, Telugu L1, completely fluent and intelligible English. Our v1 scored him "poor" because of the /v/-/w/ merger and devoiced initial stops. He nearly churned. We pulled his clips, confirmed every flagged "error" was a documented Telugu-English feature, and built the filter. His next session scored him as the strong speaker he is — and surfaced the one genuine cluster issue actually worth practising. We went deeper on the Hindi side of this work in TalkDrill's Hindi pronunciation lexicon and 31% WER reduction, and the broader accent-fairness story continued in scoring Indian-English without punishing regional accents. Try it on TalkDrill. This is the kind of speech-AI work our AI and automation team ships, proven on TalkDrill.
## Frequently Asked Questions
### Does TalkDrill try to make Indians sound American?
No. The goal is intelligible, confident English, not a foreign accent. We accept systematic mother-tongue features like retroflex consonants and the /v/-/w/ merger, and only coach substitutions that actually stop a listener from understanding the word.
### How does the scorer know my mother tongue?
You tell us during onboarding. We use that to load the right L1 substitution profile, so a Tamil speaker and a Punjabi speaker get different "accepted feature" rules. Without it, the scorer would apply a generic, less fair baseline.
### What is an L1 transfer feature versus an error?
An L1 transfer feature is a sound pattern your mother tongue systematically carries into English — predictable and shared across speakers of that language. An error is a one-off wrong sound you get right elsewhere. We score the latter and accept the former.
### How accurate is the scoring?
We calibrated against 3,400 human-rated clips and reached 0.81 agreement with trained Indian-English raters before shipping. We specifically minimise false-negative coaching — flagging an accent feature as an error — because that is the failure that erodes trust.
### Why score against intelligibility instead of a native accent?
Because intelligibility is the actual goal of speaking English in India, and a native-accent target encodes bias against every Indian L1. A heavy but clear accent communicates perfectly; our score reflects that rather than punishing it.
### Can you build accent-fair speech scoring for our app?
Yes. The pattern — phoneme-level scoring plus an L1-aware filter calibrated against human raters — transfers to any speech-assessment product. We have shipped it for English fluency and can adapt the substitution tables to your target languages.
Building a speech or pronunciation feature for Indian users?
We build accent-fair speech-scoring pipelines in 4-8 weeks. Typical cost: ₹8L-₹18L for a calibrated production v1 with an L1-aware scorer and human-review tooling. Suitable if a generic scorer would punish your users for their accent. No slides — just your audio and our honest take. Reach us at contact@softechinfra.com.
Book a 20-min CallTags:
TalkDrillSpeech AIPronunciationIndian EnglishPhoneticsVoice AIVernacular AI
