Latency Optimization in Real-Time Voice Agent Pipelines
Fast components don't guarantee a fast system when they run in sequence.
Sub-500ms latency in a voice agent pipeline is achievable. It is also fragile, expensive, and rarer in production than most vendor pages let on. Human conversation tolerates something like 300 to 500 milliseconds of gap before a response starts to feel off, and that window is the entire budget the pipeline has to work with, not a suggestion. Below 300ms and it reads as human. Between 300 and 600 it reads as slow but forgivable. Past 600, callers start mentally reverting to touch-tone systems, and somewhere north of a second and a half, they just hang up. So here's the puzzle worth sitting with: individual components already clear this bar with room to spare, and yet most deployed systems don't. ElevenLabs Flash runs TTS at roughly 75ms. Fast STT engines land around 150ms. Add those up with a generous LLM call and you're nowhere near the 1.4 to 1.7 second medians that show up across millions of real calls, with p99 stretching into multiple seconds. Fast parts, slow system. That gap is what this piece is actually about.
How the full pipeline accumulates delay before the caller hears a word
Trace the signal path and the mystery mostly resolves itself. Audio comes in over the network and through SIP signaling, gets evaluated by a voice activity detector to figure out whether the caller has stopped talking, gets transcribed, gets handed to an LLM for a response, gets synthesized back into speech, and travels back out over the network to the caller's ear. Six stages, and by default, every one of them waits for the one before it to finish.
Vendor benchmarks tend to leave this part out. STT alone can run anywhere from 100 to 500ms depending on model and load. LLM inference adds another 200 to 400ms in the typical case. TTS contributes 60 to 250ms. None of that counts the signaling overhead on either end of the call, which is real and non-zero on PSTN. Stack the floors of each stage sequentially and a 450ms end-to-end system stops looking like a component spec and starts looking like an architectural achievement, one that only holds up when streaming runs at every stage, when regions are co-located, when model contexts are pre-warmed before the call starts, and when someone is actually watching the numbers. Skip any one of those and the number drifts upward fast.
Two metrics matter here more than the rest. Time to first byte, or TTFB, measures how long it takes from when the caller starts speaking to when the system produces its first partial transcript; under 300ms is the target. Real-time factor, or RTF, is processing time divided by audio duration; keep it at 0.5 or below and the system is comfortably ahead of the incoming audio. Push past 1.0 and the pipeline is accumulating debt on every single turn, the audio equivalent of a credit card balance that never gets paid down.
Cascaded pipelines versus speech-to-speech models — which architecture to choose and when
Two architectures compete for this job in 2025. Cascaded pipelines chain separate STT, LLM, and TTS stages together. Speech-to-speech, or S2S, models collapse all three into one model that takes audio in and produces audio out directly.
S2S wins on raw latency and on preserving the texture of speech, the pauses, the emotion, the prosody that gets flattened when you route everything through a text transcript in the middle. It loses on availability, since fewer providers offer production-grade S2S, on compute cost, and on control, since you can't easily insert a tool call or an auditing step into a model that doesn't expose intermediate text. Cascaded pipelines carry a structural tax: nothing gets spoken until transcription finishes and the LLM has started producing a response, full stop, that's the sequence. In exchange, cascaded systems can be audited, composed, and have tool calls bolted onto them without surgery.
Telephony makes the decision for you more often than architecture preference does. Phone networks run at 8kHz narrowband audio, which strips out most of the acoustic detail S2S models are built to preserve. Feed a phone call into an S2S model and you're paying its compute premium for a quality advantage the phone line has already erased. Cascaded remains the standard for anything running over SIP trunks for that reason. The sensible middle path is a hybrid: S2S for quick, low-stakes conversational turns, cascaded for anything involving multi-step reasoning, tool calls, or a requirement that someone can later reconstruct what the system did and why. For most enterprise and commercial voice deployments right now, cascaded streaming is still the correct default, and the rest of this piece is about optimizing inside that constraint rather than arguing against it.
Endpointing and VAD: the latency cost hiding before transcription starts
Ask any team that's shipped a voice agent where their latency actually went, and endpointing is usually the honest answer, ahead of the LLM everyone blames first. Endpointing is the system deciding the caller has finished talking. Set that silence threshold to a conservative 700ms and you've added 700ms to every single turn before transcription even starts. It's a tax nobody notices until they measure it, and it's large enough to sink an otherwise well-tuned pipeline on its own.
Barge-in detection under 100ms is table stakes in 2025 production systems, and getting there takes real VAD tuning plus a proper interruption model, more than just dialing the silence timer down and hoping. One trick worth naming: instead of waiting out the full silence window, flush the audio buffer the instant VAD detects a pause. That alone can cut endpointing cost from around 500ms down to roughly 125ms, which is the difference between a system that feels alert and one that feels like it's dozing between sentences.
Barge-in is the mirror problem. Caller interrupts, system needs to shut up immediately, and if it doesn't, the agent starts to feel combative, like it's talking over you at a dinner party. That perceived rudeness stacks on top of whatever raw latency you've already got, so it reads as worse than the millisecond count alone would suggest. Treat the endpointing timeout as a latency parameter you tune deliberately, rather than a default you inherit from a config file, because it compounds with every stage that follows it.
Streaming STT and what "partial transcription" actually buys the pipeline
Streaming STT returns transcript fragments while the caller is still talking, sparing everyone the wait for a finished block of text once the audio stops. ElevenLabs Scribe v2 Realtime, for instance, returns partials around 150ms and streams audio in chunks, which means downstream stages can start chewing on the transcript before endpointing even fires.
That's the actual value proposition, and it's architectural, not cosmetic: the LLM can start reasoning about what the caller probably means before the last word lands. Stages that would otherwise run in strict sequence start to overlap instead. RTF is the guardrail that tells you whether this is working; cross 1.0 and you're falling behind the live audio stream, accumulating a small debt on every turn that eventually shows up as a caller talking to dead air. Streaming STT is the piece that unlocks parallelization for everything downstream of it. Skip it and every later optimization is capped by how long full transcription takes, no matter how clever the LLM or TTS setup is.
Reducing LLM inference time: speculative generation, streaming output, and prompt discipline
The LLM call eats 200 to 400ms of the budget in the typical case, and it's usually the single largest stage, which makes it the obvious target and also the hardest one to shrink through model choice alone.
Streaming the LLM's output is the foundational fix: push tokens to TTS as they're generated instead of waiting for the full response to land. That alone turns two sequential stages into one overlapping one. Some systems go further with speculative generation, running a background reasoning pass that starts while the caller is still mid-sentence, so a draft response is already forming by the time transcription wraps up, a "thinking while listening" approach. Retrieval-augmented systems can apply the same trick to knowledge lookups, firing off a retrieval call partway through the incoming stream rather than waiting for the full query text.
Prompt discipline matters more than people give it credit for. A bloated system prompt or an overstuffed context window inflates time-to-first-token in ways that have nothing to do with model quality; keeping context lean and pre-warming it before the call starts are engineering decisions, not model-selection decisions. Model size is a real lever too, and a blunt one: smaller models cut time-to-first-token at the cost of reasoning depth, which is exactly the hybrid logic from the architecture section again, route the easy turns to something fast and save the heavier model for the turns that need it.
TTS model latency benchmarks and what the numbers reveal about production tradeoffs
Line the 2025-2026 numbers up and the field looks tight at the top. Mistral's Voxtral TTS clocks 70ms TTFB with an RTF near 9.7x. Qwen's Qwen3-TTS-Flash comes in at 97ms TTFB with a mean opinion score of 4.78. Some leading TTS models sit around 75ms. Gemini 2.5 Flash runs 192ms TTFB but covers 45-plus languages with instruction-following near 97.2%.
Here's the catch, and it's the same catch as the STT and LLM numbers: that 75ms figure is model latency only. It doesn't include endpointing, transcription, LLM inference, application logic, telephony handoff, or the network hop back to the caller. No single TTS benchmark number is the end-to-end latency, and treating it as one is how teams end up surprised by their own production numbers.
What actually matters beyond the headline TTFB: when does the first audio chunk really arrive, are the chunks consistent in size (irregular chunking produces audible glitches that sound like a bad phone connection), and does the model hold up once you throw concurrent load at it instead of a single clean test call. There's a real quality-latency tradeoff at this stage; higher-fidelity models generally cost more TTFB. Over PSTN's 8kHz ceiling, though, a chunk of that fidelity advantage is inaudible anyway, which makes the faster model the rational pick for telephony specifically. That's part of why ElevenLabs Flash v2.5, alongside Cartesia Sonic-3, keeps showing up as the recommended default in open-source stack guidance, a pick driven by clearing both the latency bar and the quality bar at once rather than by being the only fast option on the market.
Network topology, co-location, and the infrastructure decisions that compound or cancel savings
Every network hop between stages, STT service, LLM endpoint, TTS engine, adds tens to hundreds of milliseconds that never show up in a component benchmark and always show up in a production bill. Co-locating STT, LLM, and TTS in the same cloud region removes the cross-region hop entirely, and it's one of the named preconditions for hitting 450ms in practice, not an optional nice-to-have.
SIP and PSTN signaling adds its own overhead before any model even sees the audio. Telephony deployments need to budget for that explicitly instead of assuming it rounds to zero, because it doesn't. Framework overhead, by contrast, is genuinely small: LiveKit's Go-based SFU hands packets to Python workers in under 2ms, and Pipecat's frame passing adds under 5ms. The latency problem tends to live in how the pipeline around the framework got built, more than in the framework itself.
Keeping connections alive between calls rather than opening fresh ones every time, and pre-warming model contexts, are the kind of unglamorous infrastructure habits that don't move your median much but noticeably tighten your p99. And p99 is the number that matters more than people admit. Optimizing the median is the baseline expectation; a p99 running into multiple seconds is where callers actually hang up, and that's a tail-latency problem, best addressed with resilience and warm-path design rather than a faster median alone.
What platform benchmarks under real conditions reveal about the gap between spec and production
Telnyx ran one of the more rigorous public tests of this: six voice AI platforms, 100 concurrent calls over real PSTN circuits, identical scripts, voice-to-voice round trip measured at p95. That's a meaningfully different exercise than a single demo call in a quiet office.
The spread in results tells its own story. Telnyx measured 71ms at p50 and 118ms at p95 in a three-carrier comparison. In a separate four-platform test running 500 production calls, Retell measured a 680ms median with 920ms at p95. That's a wide gap, and it reflects architectural choices as much as it reflects model choices; two systems built on similarly fast components can land in very different places depending on how well the stages overlap. Sub-600ms is the commonly cited bar for natural conversation in enterprise deployments, and Retell's 680ms median sits just outside it, close enough to show how thin the margin is and how much a single architectural decision moves the needle.
Published production medians running well above the so-called 450ms gold standard say something plain: this isn't solved at scale yet, and it isn't a product feature you buy off a shelf. It's an ongoing engineering discipline. Read benchmarks with that in mind. A single clean call in a demo and a p95 across 100 concurrent PSTN calls are two different measurements wearing the same units, and anyone evaluating a platform should ask for the second one before they trust the first.
Open-source frameworks versus managed platforms: the build-versus-buy calculus at scale
Two open-source frameworks dominate the orchestration layer right now. LiveKit Agents hit its 1.0 release in April 2025 and has moved through the 1.6.x line since, adding adaptive interruption handling and native MCP tool support along the way. Pipecat counters with over 100 integrations and the strongest story for embedded and local-model deployments, largely through its Ollama support.
Framework overhead, as covered above, isn't the deciding factor between them; under 2ms and under 5ms respectively is noise compared to how the surrounding pipeline gets designed. The real decision is economic. Cross roughly 10,000 minutes a month and the open-source path tends to undercut managed platforms by 60 to 80% per call. Stay under that threshold and managed is usually cheaper once you count engineering time honestly, which people frequently don't. Hamming's analysis of over 4 million production calls found that roughly half of teams on managed platforms migrate to an open-source framework within 12 months of crossing that 10,000-minute line, which is a fairly clean signal about where the cost curve actually bends.
If you're building a voice-only agent from scratch in 2026, the default stack worth starting from is LiveKit Agents or Pipecat for orchestration, a low-latency streaming STT service at the Nova-3-equivalent tier, and ElevenLabs Flash v2.5 or Cartesia Sonic-3 for synthesis, with the LLM chosen against your specific reasoning needs and latency budget rather than picked by reputation. Managed platforms still earn their keep for teams without dedicated voice infrastructure engineers, or for anyone still validating whether the product needs voice at all before they build a team around it; they get to a working system faster and absorb the operational load.
Observability and measurement as the ongoing discipline behind latency control
The 450ms target holds up only under specific conditions, and it's worth restating them plainly: streaming active at every stage, infrastructure co-located, contexts pre-warmed, and someone measuring continuously. Remove one and the number drifts, quietly at first, then not.
Instrument everything worth instrumenting: per-stage TTFB, RTF per turn, endpointing duration, LLM time-to-first-token, TTS first-chunk latency, and end-to-end voice-to-voice latency at p50, p95, and p99. Measuring only the average is close to measuring nothing, since it's the tail that decides whether callers hang up, and averages are specifically designed to hide tails.
Regressions sneak in quietly, too. A model upgrade, a slightly longer context window, a new tool integration bolted onto the LLM call, any of these can add 200ms without anyone noticing until a customer complains. Catching that requires continuous monitoring against a known baseline, rather than a benchmark run once at launch and filed away. Load testing at realistic concurrency, 100 simultaneous calls rather than one clean call in a demo, is what separates a component benchmark from something you can actually put in front of customers.
None of this changes the biological fact the whole piece started with: 300 to 500ms is still the threshold human conversation runs on, and that number isn't moving no matter how good the models get. Every component upgrade, every prompt change, every new region added to the infrastructure needs to get re-validated against that same fixed target, not assumed safe because it passed once at launch.
