Real-Time STT for Voice Agent Applications

Voice agents fail or succeed on a single question: does the system respond at the right moment, fast enough and correctly enough that the exchange feels like talking to a person instead of waiting on a fax machine. That's not a transcription problem, even though transcription sits at the center of it. Speech-to-text feeds a language model, which feeds a text-to-speech engine, and each handoff either preserves the illusion of conversation or exposes the machinery underneath, which is the gap ElevenLabs's conversational voice agents are built around closing. The global voice AI agents market is projected to grow from $2.4 billion in 2024 to $47.5 billion by 2034, a 34.8% compound annual growth rate, which means the architecture choices covered here aren't academic. They're the difference between a product that ships and one that gets abandoned mid-call.

The latency budget every voice agent lives inside

Human conversation runs on a tight clock. Responses inside 300 to 500 milliseconds feel natural; push past roughly 1.2 seconds and people start talking over the agent or just hang up. That's not a design preference, it's how turn-taking works in ordinary speech, and a voice agent that ignores it will always feel a beat behind.

The trouble is that the full pipeline eats time at every stage. Speech-to-text runs 100 to 500 milliseconds. The language model takes 350 milliseconds to a full second or more, depending on complexity. Text-to-speech adds another 75 to 200 milliseconds. Stack in network hops and orchestration overhead, and even a well-built system lands at 800 milliseconds to two seconds in practice. One illustrative breakdown from research on the pipeline: 40 milliseconds to capture the microphone input, 300 milliseconds to transcribe it, 400 milliseconds for the model to reason, 150 milliseconds to speak the answer. Individually those numbers look fine. Added together, they don't.

That's the uncomfortable math of latency: it compounds rather than adding cleanly, because each stage has to wait on the one before it. A transcription layer that returns results in around 150 milliseconds and a synthesis layer that speaks in 75 milliseconds can still leave a system north of 800 milliseconds end to end, because nothing overlaps unless it's built to. Industry benchmarks have identified sub-600 milliseconds as the bar for something to feel conversational rather than transactional.

Given that math, speech-to-text can't claim the whole budget for itself. It has to leave enough room for the model and the voice synthesis to do their jobs without blowing past the ceiling. Meeting that constraint means the industry has largely converged on one answer: stop waiting for the speaker to finish, and start processing the audio while it's still coming in.

How Voice Activity Detection shapes turn-taking before STT even runs

Before any word gets transcribed, something has to decide when a person started talking and when they stopped. That's Voice Activity Detection, and getting it wrong produces the two failure modes that annoy users most: the agent cuts someone off mid-sentence, or it sits there in silence long after the person finished, waiting for a signal that already came and went.

Silero VAD has become a common default in open-source voice pipelines, processing audio in under a millisecond per chunk on ordinary CPU hardware, with a footprint of only 1 to 2 megabytes and an MIT license that keeps it out of legal review purgatory. PipeCat's smart-turn-v2 model is showing up increasingly in cloud orchestration setups, including deployments on Cloudflare Workers AI.

Worth separating: VAD detects that someone is speaking. Endpointing decides that they've finished their turn. Those sound like the same job, but conflating them is exactly how systems end up interrupting people who just paused to think. Endpointing is tunable, and the tuning is a real tradeoff rather than a settings toggle with one correct position. Aggressive endpointing shaves off latency but risks cutting speakers off before they're done. Conservative endpointing waits longer to be sure, which guarantees completeness at the cost of a laggy-feeling agent. There's no universal answer, only a fit for the use case.

And none of this is recoverable downstream. If the VAD hands the speech-to-text model a bad audio segment, chopped short or padded with dead air, the transcript comes out wrong no matter how good the underlying model is. Garbage in, garbage out is not a cliché here, it's the literal signal path.

Accuracy in real conditions: what WER actually measures and what it misses

Word Error Rate is the industry's go-to accuracy benchmark: count the substitutions, deletions, and insertions relative to the correct transcript, divide by the total word count, and lower is better. Among current streaming models, AssemblyAI's Universal-2 posts a WER of 14.5%, the strongest accuracy figure among streaming systems in the benchmarks, while Deepgram's Nova-3 comes in at 18.3%. That gap sounds small until the transcript feeds a medical intake form or a financial services compliance check, where a single misheard word changes the meaning of the record.

WER numbers from clean, controlled benchmark audio tell you almost nothing about how a model behaves on an actual customer service call, with background noise, two people talking over each other, or a caller whose accent the training data barely represented. Production audio is messy in ways that benchmark audio, by design, is not.

There's also a distinction that WER doesn't capture at all: getting most of the individual words right while still mangling the sentence's meaning. A model can nail ninety percent of the words in an utterance and still hand the downstream language model a sentence that means the opposite of what the caller said. For a voice agent that has to act on what it heard, that's a failure WER simply doesn't register.

Domain vocabulary compounds the problem. Product names, medical terms, or industry-specific jargon that the training data barely touched will spike error rates in ways a generic benchmark won't predict. The practical fix is keyterm prompting, feeding the model specific words or phrases ahead of time so it knows to expect them, without retraining anything. As one framing in the industry has it, the best model for a voice agent isn't whichever one tops the leaderboard on clean audio. It's the one that behaves predictably when the audio isn't clean, which is most of the time.

Streaming architecture: why batch transcription cannot serve conversational agents

Batch transcription, the kind that processes a complete audio file after the fact, is fine for meeting notes and useless for live conversation. Whisper running self-hosted in a standard batch configuration takes 1 to 5 seconds to return a transcript. That alone rules out any chance of hitting a sub-600 millisecond response window; the transcription step by itself would already have blown the whole budget. An optimized variant, WhisperX, gets that down to 380 to 520 milliseconds, which is a real improvement but still eats most of the latency budget before the language model has said a word.

Streaming architecture works differently. It emits partial, interim transcripts continuously as audio comes in, rather than waiting for the speaker to stop. That means the language model can start reasoning about what's being said before the sentence is finished, recovering time that a batch system simply cannot get back. The pattern goes like this: partial transcripts feed speculative execution in the language model, which starts generating a likely response early; the final transcript then either confirms that guess or throws it out and starts over. That speculative approach is a big part of how sub-500 millisecond end-to-end responses are possible at all, even when each individual component takes hundreds of milliseconds on its own.

Streaming also opens up capabilities that batch transcription structurally cannot support, enabling downstream systems to act on what's being said while the call is still happening, not in a report generated after the customer has hung up. Scribe v2 Realtime is built around this streaming-first approach, returning transcripts in under 150 milliseconds across more than 90 languages, aimed specifically at agent and meeting contexts where those interim results are the whole point.

Multilingual and accent coverage as a production requirement, not a feature flag

Language support numbers vary widely by provider, and the headline count is the least useful number in the spec sheet. Google Cloud Speech-to-Text lists support for more than 125 languages. Scribe v2 Realtime covers more than 90. Some streaming providers cover 45-plus. None of that tells you whether the model actually performs well on the specific accent a caller happens to have.

Coverage and quality are different claims, and treating them as the same one is a common way deployments go sideways. A model that technically supports English may still struggle badly with Indian English or Nigerian English if the training data skewed toward American or British speech patterns. The same goes for regional Spanish variants: es-419 is not es-ES, and a model tuned on one won't necessarily transfer cleanly to the other.

Some systems now handle a caller switching languages mid-sentence, which matters more than it sounds like it should for bilingual households and border regions. Some systems handle mid-sentence switching natively across ten or more languages including English, Spanish, French, German, Hindi, Russian, Portuguese, Japanese, Italian, and Dutch. Separately, some providers return not just a transcript but a classification of the speaker's regional accent, distinguishing en-US from en-GB from en-IN, which lets a downstream system route the call or personalize the response instead of treating every English speaker as identical.

Architecturally, single-model multilingual systems keep latency steady across languages because there's no handoff between separate models tuned for separate languages. Multi-model setups introduce a routing decision at the start of every call, and that routing step is itself a source of delay and a place where something can go wrong. For regulated industries operating across borders, data residency stacks another constraint on top of all this: healthcare and finance deployments in particular need to know where the audio and the transcript actually live, and some providers offer regional deployment options built around that requirement.

Comparing the major real-time STT options for voice agent deployments

No provider wins on every axis, so the comparison is really about which tradeoff fits the deployment.

Some streaming providers post roughly 150 milliseconds to first token in the US, a WER around 18.3%, coverage across 45-plus languages, and pricing around $0.0043 per minute, with bundled voice agent packages that wrap speech-to-text, the language model, and text-to-speech together. That class of option suits deployments where latency is the constraint that can't move.

AssemblyAI's Universal-2 runs 300 to 600 milliseconds of latency but posts the best accuracy of the streaming models at 14.5% WER, with particular strength in medical and sales contexts, at roughly $0.00025 per second. That's the accuracy-first option, worth the extra latency when a misheard word costs more than a slightly slower reply.

Google Cloud Speech-to-Text covers more than 125 languages with streaming support and compliance features suited to enterprise deployments. It is one option for organizations with enterprise compliance requirements around language breadth.

Telnyx's Real-Time STT API is built around a unified interface covering multiple underlying ASR engines behind a single API. That's a direct answer to a real problem: no single engine wins on speed, accuracy, and language breadth simultaneously, so a unified API saves a team from integrating several vendors separately.

Whisper, self-hosted, still delivers the strongest offline accuracy of the group, but at a cost. Standard batch mode runs 1 to 5 seconds, and the optimized WhisperX variant gets to 380 to 520 milliseconds, still near the outer edge of what a real-time budget can tolerate. It fits hybrid setups that mix live conversation with batch analytics, or deployments where data has to stay on-premise no matter what.

xAI has announced a standalone speech-to-text API built on the same technology stack as its Grok Voice product, already in use in Tesla vehicles and Starlink customer support. It's being positioned for voice agents, real-time transcription, and accessibility use cases, but specific latency and language coverage numbers hadn't been published at the time of this reporting.

Orchestration: how the components around STT determine whether the pipeline holds

None of the speed or accuracy numbers above matter if the layer coordinating them falls apart. Orchestration manages the real-time handoffs between speech-to-text, the language model, and speech synthesis: who's turn it is, what happens when someone interrupts, how session state survives a hiccup, what the system does when something errors out mid-call. A great transcription model wired into sloppy orchestration still produces a broken conversation.

A few patterns show up repeatedly in systems that hold together. Streaming generation lets text-to-speech start speaking as the language model's tokens arrive, rather than waiting for the full response to finish generating. Speculative execution lets the model start reasoning on a partial, unconfirmed transcript and throws the guess away if the final transcript comes back different. Model routing sends simple requests to a fast, cheap model and routes complex ones to something more capable, so the system isn't paying the latency cost of a heavyweight model on every single turn.

Interruption handling belongs squarely in this layer, and it's really an extension of the VAD problem from earlier. When a caller talks over the agent, the system has to notice the new speech immediately, cut the text-to-speech output, and loop back through speech-to-text and the language model without losing track of what the conversation was about a moment ago. Weak VAD integration at the orchestration level is exactly where that loop breaks.

Cloudflare's Realtime Agents platform is a useful illustration of what edge deployment buys here: it runs voice AI pipelines across its global network, using WebSocket-based inference and Deepgram's models running on Workers AI for speech-to-text and synthesis across more than 330 cities. Getting the compute physically closer to the caller trims network latency in a way that no amount of tuning at the model level can substitute for. On the framework side, open-source projects like PipeCat provide the scaffolding that connects a chosen speech-to-text provider over WebSocket or SDK, and the framework choice quietly determines which providers are easy to plug in and which require custom integration work.

What to optimize for when the requirements actually conflict

Here's the tension underneath everything above: no speech-to-text provider today wins simultaneously on the lowest latency, the highest accuracy, and the broadest language coverage. The practical reality of the market gets at this directly, and it's the reason multi-vendor architectures keep showing up despite the extra integration work they demand.

When latency is the binding constraint, outbound sales calls, IVR systems, live agent assist, the right move is to optimize for time-to-first-token and lean on streaming architecture, accepting a somewhat higher WER if the vocabulary in play is predictable enough to handle with keyterm prompting.

When accuracy is the binding constraint, medical intake, legal dictation, financial services, it makes more sense to accept the 300 to 600 millisecond latency of a model like AssemblyAI's Universal-2 and claw back overall response time through the language model and speech synthesis stages instead of compromising on transcription quality.

When language coverage is the constraint, global deployments serving multilingual populations, a single-model multilingual system avoids the routing complexity and the latency spikes that show up at language boundaries in multi-model setups. And the evaluation there has to happen per language in the actual target market, not off an aggregate coverage count on a spec sheet.

Regulated deployments add a filter before any of this even starts: data residency requirements and certifications like SOC 2, HIPAA, or GDPR narrow the field of viable providers before latency or accuracy enters the conversation at all.

There's also a structural argument worth stating plainly: every seam between vendors in a fragmented stack, separate providers for speech-to-text, the language model, and speech synthesis, adds its own slice of integration latency. Where a single provider covers the full path from speech-to-text through synthesis with components built to work together, the orchestration overhead shrinks and the end-to-end latency becomes far more predictable.

None of this is a decision made once and forgotten. Model accuracy improves, latency benchmarks shift, and the tradeoff frontier moves every few months in this space. Building the orchestration layer with enough abstraction that a speech-to-text provider can be swapped out without rebuilding the agent's underlying logic isn't a nice-to-have. It's the only way to keep up without starting over every time the landscape changes.