STT Accuracy Benchmarks Across Accents and Dialects

Word Error Rate sounds like a clean number. It isn't. WER counts substitutions, insertions, and deletions, adds them up, and divides by the total words in the reference transcript, times 100. That's the whole formula. What it hides is which words got missed, whose voice caused the miss, and whether the mistake mattered at all downstream. A vendor's homepage says 95-98% accuracy and moves on, and that figure was almost certainly measured on read speech, in a quiet room, by someone who speaks the exact English dialect the model trained on. Almost nobody's actual deployment looks like that.

What WER actually counts, and what it silently ignores

WER treats every mistake the same, trivial or catastrophic. Drop a filler word like "um" and it's one error. Drop the number "15" from "room 15" and it's also one error. Same penalty, wildly different consequences. In a voice-agent pipeline, such as those built on conversational voice platforms like ElevenLabs, if the model transcribes "cannot" as "can't," that logs as a substitution error even though the language model downstream understands both identically. The metric just punished a distinction nobody downstream cares about.

Now flip it. A deletion, a word that vanishes entirely, can undermine an agent's ability to respond correctly. The interaction may break in ways that are invisible in the aggregate score. WER scores that failure exactly the same as it scores "cannot" versus "can't," despite one being invisible and the other breaking the interaction. The metric can't tell a shrug from a system crash.

WER also collapses every speaker into one aggregate score, so a model can post a great overall number while quietly failing a specific accent group at a much higher rate. Nobody reading the headline figure would know.

Other metrics fill some of these gaps, though none of them fix WER, they just measure something WER can't see. Character Error Rate matters for languages without clean word boundaries. cpWER handles multi-speaker audio, penalizing both the transcription and who-said-what attribution. Domain keyword WER or missed-entity rate tracks whether the model caught the terms that actually matter: drug names in a clinical note, case numbers in a legal filing. Semantic WER skips word-matching entirely and asks a reasoning model to judge whether the meaning survived. Pipecat's open-source STT benchmark, referenced in AssemblyAI's 2026 accuracy guide, pushes this approach specifically to cut the scoring bias built into strict word-matching.

The documented accuracy gaps across accents, dialects, and speaker demographics

The research here is not ambiguous. Koenecke et al., published in ACM CHI 2025 proceedings, tested five commercial ASR systems and found an average error rate of 35% for Black speakers against 19% for white speakers, a gap that held even when both groups read identical scripted phrases. The researchers traced the cause to acoustic factors, not vocabulary or grammar. Same words, same script, worse recognition. That's not a training-data quirk that fixes itself quietly over time. That's a system built around one acoustic profile and sold as universal.

Dubois et al. (2024) found mainstream videoconferencing and social platforms produce roughly double the error rate for L2 English speakers compared to native speakers, and L2 speakers make up a large share of English speakers on the planet. Not a fringe case being poorly served. The majority use case, underperforming, while most vendor pitches skip straight past it.

Research has found Whisper favors certain English varieties over others, with accuracy dropping further for non-native accents, tied to differences in prosody, vowels, and the gap between spontaneous and scripted speech. McGuire's 2025 work on the L2-ARCTIC corpus, covering Arabic, Chinese, Hindi, Korean, Spanish, and Vietnamese first-language speakers, found Whisper and AssemblyAI both landing close to human-level accuracy on read speech, mean match error rates of 0.054 and 0.056. Encouraging, until you remember read speech is the easy case. Real deployments involve people talking, not reciting off a script.

Gender bias shows up too, but it doesn't point one direction, which is the more useful finding. Some studies have found ASR systems performing worse on female speakers. Other work has found the opposite pattern in certain models: worse performance on male speech. The bias is real both times. It just isn't consistent, so nobody gets to assume their model is safe on this axis without testing it directly.

Koźmińska, writing in the Journal of Sociolinguistics in 2025, coined the idea of accents becoming "hyperaudible": ASR failure doesn't just produce a bad transcript, it can reinforce existing social and regional disparities every time it happens. Global audits of ASR systems have found the worst error rates concentrated among speakers from underrepresented regions.

Line all of this up and the pattern isn't subtle: the speaker populations most absent from training data pay the steepest accuracy tax the moment the system goes live. That's the training data working exactly as designed, for a narrower population than the marketing implies.

Why standard benchmark datasets cannot reveal these gaps

The Open ASR Leaderboard, a community-maintained benchmark, reports WER and real-time factor across roughly a dozen datasets including LibriSpeech, TED-LIUM 3, GigaSpeech, VoxPopuli, AMI, Earnings22, SPGISpeech, and Common Voice. Useful project. Wrong lens for this problem. Research points out these sets skew heavily North American and British English, so a model can top the leaderboard while offering almost no signal on African-accented English.

The training data explains most of this on its own. Common Voice's English corpus runs past 2,000 hours of recorded speech. Its Arabic corpus sits at 89 hours. That imbalance maps almost directly onto the accuracy gap: languages with deep training corpora (English, Spanish, and similar high-resource languages) land in the 3.5% to 6.8% WER range, while data-poor languages like Lithuanian, Latvian, and Icelandic sit at 12% to 18%. Mozilla's own Common Voice Accent Analysis found this correlation between training volume and accuracy, not a loose trend worth shrugging off.

Multilingual benchmarks compound the problem by chasing language count over acoustic variety. A test set stuffed with clean, prompted, single-speaker audio in forty languages tells you almost nothing about overlapping speakers, background noise, or a phone call picked up on a bad connection. Low WER on these leaderboards increasingly measures how well a model adapted to the test distribution, not how well it holds up against an actual human talking in an actual room. Middle East and Southeast Asia alone represent over a billion speakers who stay thin on the ground in any current standard benchmark, a strange thing to say about a "standard."

What GigaSpeechBench reveals when the test conditions match real-world audio

GigaSpeechBench (arxiv.org/abs/2606.28884) built its test set specifically to close that gap: 680 hours of human-annotated, in-the-wild speech, split across five modules covering 12 low-resource Middle Eastern and Southeast Asian languages plus Japanese and Korean, six Chinese dialects (Xiang, Jin, Gan, Min, Yue, Wu), six English accents, dense vocabulary from 12 professional domains, and speech from older adults and children.

The six English accents tested are Chinese-accented, Indonesian-accented, Japanese-accented, Philippine-accented, Scottish, and Singapore English. Scottish English turned out to be the hardest accent in the entire set, for nearly every model, with WER spanning a wide range across models tested. A native English accent breaking models harder than most non-native ones should embarrass anyone still calling their training data "English-language audio" without saying which English.

Indonesian-accented English produced the widest spread across vendors: Azure at 33.00%, Google Chirp 3 at 8.11%, Scribe v2 at 11.76%, with other models spread across the range between those two. No model wins across the board. A system strong on Indonesian-accented English might land in the middle of the pack on Scottish English, so accent coverage can't be read off a single number. It has to get checked accent by accent, every time, or it doesn't get checked at all.

Arabic dialect results in the same benchmark show how much variation hides inside "one language." Results varied widely across Arabic dialects on the same system, with some dialects producing dramatically higher error rates than others on the same platform. A large gap between the easiest and hardest dialect, on the same system, in the same language. "Supports Arabic" turns out to mean almost nothing without the dialect attached.

FLEURS results, covering Egyptian, Indonesian, Malay, Filipino, Vietnamese, Thai, Japanese, and Korean, show the same instability. Some models post sharply different error rates on Egyptian Arabic versus Indonesian. Others show similar swings between Egyptian Arabic and Japanese. No single model leads across all languages in this group; a model strong on one language can trail on several others. There's no shortcut past this. Per-language, per-accent inspection is the only method that tells the truth, and anyone selling a single blended accuracy number is selling a rounding error dressed up as fact.

How leading STT models are architected and where each sits on the accuracy-latency tradeoff

Whisper Large V3 runs 1.55 billion parameters, trained on 680,000 hours of multilingual, multitask web audio, covering more than 99 languages. It averages 7.4% WER on mixed benchmarks and does its best work on high-resource languages: English, Spanish, French, German. Whisper Large V3 Turbo trims that down to 809 million parameters by cutting the decoder from 32 layers to 4, running roughly six times faster with accuracy within a point or two of the full model on most languages. The gap widens on lower-resource languages like Thai, where the smaller model gives up more ground than it does elsewhere. Speed costs the most exactly where the language already had the least training data to spare, and that's the real tradeoff, not the abstract one vendors advertise.

Streaming and batch are not the same job, and treating them as interchangeable is where most procurement decisions go wrong. Independent AA-WER testing puts Nova-3 around 18% WER on mixed real-world data with latency under 300 milliseconds; its maker's own published benchmark claims a 54.2% WER reduction against the nearest competitor, a figure worth treating as a vendor claim rather than a neutral result. AssemblyAI's Universal-3.5 Pro and its real-time counterpart sit at the top of that company's own 2026 lineup, covering 18 languages with mid-sentence code-switching support.

Batch-transcription models chase accuracy and can afford to think. Streaming models get built for speed and have to answer before the sentence even finishes. In a voice-agent pipeline, the entire round trip, speech-to-text, language model reasoning, text-to-speech, needs to land around 800 milliseconds total. That leaves the STT step something like 150 to 300 milliseconds, which rules out most high-accuracy batch models no matter how good their WER looks on paper. Anyone picking a model off a leaderboard without checking which column measures speed is picking blind.

Open-source models offer full control over infrastructure and skip per-minute billing entirely. But GPU costs and the engineering headcount needed to run that infrastructure can eat the savings alive once volume climbs, a tradeoff that only shows up once someone runs the math at scale, usually after the contract's already signed.

How to design an evaluation that actually reflects your deployment population

Start with the audio, not the leaderboard. Pull samples from the actual speakers, devices, and rooms the system will face in production. A contact center in Lagos and a podcast studio in Chicago are not the same evaluation problem, and testing one to represent the other tells nobody anything useful.

Break the results out by speaker group instead of reporting one blended number. A model can look strong in aggregate while quietly failing one accent at 35% WER, buried under a healthy-looking average. Pick the metric that fits the job: Semantic WER for voice agents where meaning matters more than exact wording, cpWER for anything with multiple speakers talking over each other, domain keyword WER for medical, legal, or technical audio where missing the one term that mattered is the whole ballgame.

Separate deletions from substitutions when scoring, especially for real-time agents. A missed word that stalls a turn is a worse failure than a plausible near-miss substitution, even though standard WER scores them the same. Weight the errors to match what actually breaks the conversation, not what's easiest to compute.

For any multilingual or accented deployment, test each target language and accent pair on its own, closer to GigaSpeechBench's per-group methodology than to trusting one aggregate figure to speak for all of them. Treat every vendor-published benchmark as a starting point, not a conclusion: most of them are self-reported, on test sets chosen to flatter the model. Rerun the numbers on actual production audio before committing infrastructure budget to any of it. The STT layer sits underneath everything else in a voice pipeline, so the accuracy it delivers, or fails to deliver, becomes the ceiling for whatever gets built on top of it.