Interruption Handling and Turn-Taking Logic in Voice Agent Systems
How voice agents decide when to speak and stop speaking.
For voice agent systems to hold a natural conversation, engineers must solve two different problems: deciding when the user has finished and handling interruptions when the agent is speaking. Solving either in isolation is not enough. The system must take the floor at the right moment, yield it quickly, and preserve what remains unsaid.
The two problems at the center of every voice agent conversation
End-of-turn detection decides when the agent should start to reply. Interruption handling decides when it should stop because the user has started speaking and wants the floor. These decisions concern opposite sides of the same exchange, but they require different pipeline components and fail in different ways. An agent can be quick to answer but slow to stop, or easy to interrupt but always cutting into sentences before the user has finished. Neither qualifies as a good listener.
Human turn-taking happens in very short gaps across languages because people predict when the other speaker will finish. A voice agent must predict the same thing over a phone line, without being able to see the speaker. Most production systems also divide the exchange across speech-to-text, an LLM, and text-to-speech, so conversational timing crosses several processing boundaries. At the end of a user turn, the central decision is when to hand the transcript to the LLM. During the agent’s reply, an interruption requires stopping three simultaneous in-flight processes, which makes yielding the floor an orchestration problem involving speech detection.
Why silence duration alone is not a reliable end-of-turn signal
Silence-based endpointing treats a pause above a chosen threshold as a completed turn. That creates opposing errors because a short threshold cuts into pauses within a thought, while a long threshold delays replies after clearly finished statements. The same one-second silence can mean the user has finished, is thinking, or is reading a number from a card. Duration measures the gap but does not explain it. A stopwatch is a poor interpreter.
A setting that accommodates someone working through a number can make a straightforward exchange feel needlessly slow. Conversely, a setting chosen for brisk responses can seize the floor while the caller is still composing a sentence. What is missing is evidence about the meaning of the pause.
The cost becomes concrete in appointment scheduling. If a patient says "okay" while an agent recites available slots, a silence-based system can treat that acknowledgment as a completed interruption turn and restart the scheduling sequence. The patient must then repeat information they were ready to confirm. This failure exposes the limit of the timer approach because detecting that a sound has ended does not establish what conversational role it played.
How semantic turn-taking models fix the detection problem silence can't solve
Semantic turn-taking models add the evidence that a silence threshold lacks. They predict whether the speaker has finished using linguistic content, including sentence structure, intonation projection, and syntactic completion. A pause inside an unfinished thought can therefore remain part of the user’s turn, delaying the reply. Silence still contributes evidence, but it no longer carries the whole decision.
Production implementations in 2026 include Pipecat's LocalSmartTurnAnalyzerV3 under its Smart Turn feature, LiveKit's TurnDetector, and Vapi's endpointing controls. LiveKit's TurnDetector encodes user audio directly to capture both what is said and how it is said. These implementations place turn-taking decisions above the raw voice activity detection, or VAD, pipeline. At this layer, backchannels, barge-ins, and completed turns become learned classifications rather than conclusions drawn from an energy threshold. The defining shift is not the removal of VAD, which remains a fast first-cut filter, but the transfer of conversational judgment to a model equipped to make it.
What the interruption-handling pipeline must do within a tight time window
Interruption handling begins with a practical prerequisite that model discussions can easily obscure. Echo cancellation must prevent the agent’s own voice, played through a speaker and picked up by the microphone, from being treated as user speech. Without it, the system can interrupt itself. That is a feedback problem.
With echo addressed, the VAD pipeline provides three layers of screening. An energy threshold is fast but coarse and cannot distinguish speech from noise. A voice classifier such as Silero VAD or a custom CNN then produces a per-frame confidence score. A minimum-duration guard requires sustained voice, screening out coughs or brief sounds alongside the other speech checks, while the turn-taking layer decides whether detected speech should make the agent stop.
Once a barge-in is confirmed, the interruption-handling pipeline has a tight time target for yielding the floor. That requires coordinating cancellation of LLM generation, text-to-speech synthesis, and audio playback. TTS cancellation is particularly easy to under-engineer because a stop decision does not itself remove audio already waiting to play. The streaming TTS provider must allow mid-stream cancellation, the audio buffers must be emptied, and the WebSocket connection must be back in listening state within the target window. Detection is useful only if the rest of the system can act on it.
The difference between a true interruption and a backchannel — and why misclassifying them breaks conversations
Not every voice heard during an agent’s reply is a request to take over. A true interruption claims the floor, while a backchannel such as "uh-huh," "yeah," or "mm-hmm" signals engagement without claiming a turn. The acoustic event is speech in both cases, but its conversational function differs. Treating those functions as interchangeable makes an agent stop precisely when the user is encouraging it to continue.
This error is more disruptive than a slightly slow response to a genuine interruption because it changes the structure of the exchange. A false barge-in makes the agent abandon its thought and re-prompt, which can compound into lost context as the conversation proceeds. The scheduling loop illustrates that distinction between acknowledging information and asking for a new turn. Faster cancellation cannot repair the wrong classification and may simply execute it more efficiently. In production, a dedicated turn-taking model above raw VAD learns the boundary between backchannel and barge-in.
What the agent must remember after being interrupted
Stopping playback solves only the immediate problem of who is speaking. Recovery depends on recording what the user actually heard. If conversation history includes undelivered content as though it had been spoken, the LLM can treat that information as received and skip it later. The transcript then records intended speech as delivered speech. Preserving the boundary between those two is essential to preserving context.
Ellis et al. (2026) demonstrated the clinical cost in FAQ delivery. Competitive interruption led to provision-coverage failures in all four models in every case tested. The agent answered the new question and moved on, leaving safety-netting advice that had been interrupted without re-delivery. Responding well to the newest question was not the same as completing the task.
Recovery also has to account for work already in flight. The default tool-call policy should let idempotent reads finish so their results can be used in the next turn, but hold mutations until the interruption is resolved and the user gives explicit confirmation. Dialogue wording adds another source of variation. A brief apology marker such as "sorry to interrupt" shifts recovery by tens of percentage points, inconsistently across models.
Why benchmarks measuring aggregate interruption robustness can mislead builders
A single interruption-robustness score can conceal the failure mode that matters most to a deployment. Performance varies by interruption type, clinical phase, task cell, and model, so an overall leader can still fail systematically in a relevant condition. For builders selecting a platform, the methodological issue is what the score combines. Averaging different outcomes does not establish that any particular workflow is safe or reliable. A leaderboard is not a deployment specification.
EVA-Bench found that no evaluated system was simultaneously good at task accuracy and conversational experience. A summary score can therefore conceal which dimension a system sacrifices. Ellis et al. (2026) make the clinical objection more directly because rankings differed across history-taking and FAQ cells, outcomes varied by clinical phase, and every evaluated model failed under at least one interruption condition. Those differences make a single robustness number uninterpretable for deployment decisions.
IHBench (Salimi et al., 2026) offers a structured alternative by evaluating post-interruption recovery across 10 enterprise domains. It injects six interruption types at controlled points and scores task fulfillment and recovery quality separately. The useful comparison is consequently not just which system scores higher, but which conditions it handles and which outcomes the scoring preserves.
How regulated and high-stakes deployments change the design constraints
In regulated domains, architecture has consequences beyond conversational smoothness. The cascaded pipeline provides an auditable text record, which is a compliance asset even though it handles interruptions less gracefully than native speech-to-speech models. The choice is therefore not purely about reducing delay. It also concerns whether an interaction can be reviewed and whether its failures remain visible.
Healthcare makes the timing stakes particularly clear. With sequential processing and no interrupt detection, the system took 18–22 seconds to respond after a user tried to interrupt, a delay that poses a patient safety issue. The clinical coverage failures also show that an agent can answer an interruption and still leave out information it must give, so responsiveness alone cannot show that it is safe.
In healthcare, financial, and legal contexts, it must be possible to review every voice interaction. Interruptions must be logged with timestamps indicating who interrupted and why, and fallback to human escalation must be available at any point in the conversation. Meeting these rules calls for cell-level evaluation and a conversation history that records what happened accurately.
The latency targets and tuning levers that connect both problems into one system
End-of-turn detection and interruption handling ultimately share a latency budget, but they need separate measurements within it. The production targets include end-to-end latency under 800 ms, a turn-taking gap of a few hundred milliseconds, and a false barge-in rate below a low threshold. Once a real interruption is confirmed, time-to-stop should be well within a hundred milliseconds. That stopping target is tighter than the broader interruption-handling window. Measuring each interval separately prevents a fast component from disguising a slow exchange.
Interruption handling affects perceived latency by a factor of two to three. A system that detects interruptions and stops cleanly can feel faster than one with lower raw response latency but poor recovery. The tuning levers therefore extend beyond response generation to the VAD energy threshold and minimum-duration guard, the backchannel-versus-barge-in classification boundary, TTS cancellation speed, LLM cancel latency, and context-preservation policy. Each must be measured independently and then validated together under realistic call conditions, including noisy audio and non-native accents. Otherwise, an improvement in one layer may leave the conversational failure untouched.
Platforms exposing these controls through APIs and developer-configurable pipelines allow teams to tune individual layers. Managed services that abstract orchestration trade some of that control for faster deployment and less visibility into individual failure modes. Natural conversation is the result of those distinct mechanisms working together.
