Speech-to-Text Models With Non-Speech Audio Tagging Capabilities

Standard speech-to-text models transcribe words but go blind, or worse, hallucinate, when confronted with laughter, background noise, or environmental sounds. That's because mainstream speech recognition is built for one job: pull linguistic content out of an audio signal and throw away everything else as noise. A cough gets suppressed. A pause gets swallowed. A burst of laughter either vanishes or, in the worse case, gets mistaken for words that were never spoken.

Research presented at ICASSP 2025 gave this problem a name and a body count: Whisper, one of the most widely deployed transcription models on the planet, fabricates text when it hits non-speech audio it wasn't trained to handle. That's not a minor quality gap. A model inventing sentences because it heard a car horn is a reliability and safety issue the moment it runs in a call center, a courtroom transcription tool, or a clinical dictation system.

Real audio is never clean. Background noise, music, environmental clatter, and paralinguistic sound (the breath, the laugh, the sigh) all appear in the same waveform as the words a model is trying to extract. Non-verbal vocal events aren't clutter to filter out either: a word-only transcript discards the information they carry about a speaker's intent and emotional state. Someone saying "sure, fine" while laughing means something different than the same words said flat.

This piece covers what non-speech audio tagging actually is, how it got built into transcription systems, which platforms ship it today, and how a developer picks between them.

What non-speech audio tagging means inside an STT system

Three things get lumped together under "speech-to-text" that are actually distinct engineering problems. Pure transcription converts audio to words, full stop.

In practice, an audio event label looks like a timestamped note sitting next to the words: "laughter at 00:14 to 00:16," attached directly to the transcript rather than delivered as a disconnected report from a second pipeline.

Research out of the WESR framework draws a further line worth knowing: some non-verbal events are discrete, standing alone without overlapping speech (a cough between sentences), while others are continuous, mixed directly into speech itself (laughing while talking). A system built to catch the first type often misses the second, because continuous events require detecting a change in vocal quality mid-utterance rather than spotting a gap.

Resolution matters too. Each choice has consequences downstream. A segment-level tag is fine for a podcast summary. A word-aligned tag is what a voice agent needs if it's trying to figure out whether the user laughed before or after saying "no."

How Whisper-AT established the frozen-encoder-plus-tagging-head paradigm

Whisper-AT, introduced at Interspeech 2023, is the model that made non-speech tagging a solved engineering problem rather than an open research question. The approach: freeze the entire Whisper backbone, don't touch a single weight, and train a lightweight Time- and Layer-wise Transformer on top of the frozen encoder's internal representations to do the tagging.

The finding that made this possible is almost counterintuitive. Whisper's internal audio representation turns out not to be noise-invariant at all, it's actually highly correlated with the type of non-speech sound present, meaning Whisper has been quietly recognizing speech conditioned on background noise the entire time. That information was sitting inside the model unused. Whisper-AT just built a small head to read it out.

The payoff: Whisper-AT produces labels across 527 AudioSet categories at adjustable temporal resolution, running alongside the standard ASR transcript, for under 1% additional compute and through an API that looks identical to plain Whisper Whisper-AT / Interspeech 2023. Freezing the backbone means none of the existing transcription quality gets touched; the only new cost is training a small tagging head, which is a pattern that generalizes to any pretrained encoder. That's the real contribution here: not the specific labels, but proof that tagging can ride for free on top of a model that already exists.

It's not a finished answer, though. AudioSet's 527 categories are broad (dog bark, vehicle, music) rather than fine-grained vocal events like a sob versus a sigh Whisper-AT / Interspeech 2023. The temporal resolution is tunable but stops short of word-level alignment, and the tags live as a parallel stream next to the transcript rather than woven into it. Whisper-AT tells you something happened somewhere in a window.

Word-level event localization and the WESR framework

WESR, short for Word-level Event-Speech Recognition, published in the ACL Anthology in 2026 by Chenchen Yang and coauthors, moves from segment-level audio tags to precise localization of non-verbal events at word-aligned positions within a transcript. Instead of "laughter somewhere in this clip," the system reports laughter attached to a specific word boundary.

Three things often get conflated: pure transcription, audio event detection, and integrated audio tagging within a transcription model. First, a taxonomy of 21 vocal event categories, split between discrete and continuous types WESR / ACL Anthology 2026 WESR Table 1 Deepgram Aura-2. Second, WESR-Bench, an expert-annotated set of over 900 utterances using a position-aware scoring protocol that separates two failure modes previously tangled together: did the system miss the event, or did it detect the event but place it in the wrong spot WESR / ACL Anthology 2026 WESR Table 1 Deepgram Aura-2? Third, a baseline model trained on a 1,700+ hour corpus that beats both open-source audio-language models and commercial APIs on this task while preserving ASR quality WESR / ACL Anthology 2026 WESR Table 1 Deepgram Aura-2.

The training data behind this ecosystem shows a large jump in scale, which produces much broader category and hour coverage in later datasets. NVSpeech-170k spans 18 categories across 732 hours in English and Chinese WESR / ACL Anthology 2026 WESR Table 1. NonVerbalSpeech-38K covers 10 categories across 131 hours in English and Chinese WESR / ACL Anthology 2026 WESR Table 1. That's a meaningful jump: from 131 hours to 1,767 hours WESR / ACL Anthology 2026 WESR Table 1.

A companion effort, NVV-Locator, published in September 2026, pushes the same idea further by treating paralinguistic vocalizations as inline tokens the model decodes directly within the transcript stream, rather than as a separate annotation layer. The position-aware benchmark matters more than it sounds like it should: before WESR-Bench, nobody could tell whether a system's poor score meant it missed events entirely or just misplaced them, which made comparing systems close to meaningless.

None of this ships yet. WESR is a research framework, not yet a shipping product; the gap between benchmark conditions and production audio diversity is still large.

Commercial STT platforms that ship non-speech audio event tagging today

A handful of commercial APIs have actually shipped something in this space as a documented production feature, and they've taken noticeably different roads to get there.

AssemblyAI's Universal-3.5 Pro and Universal-Streaming models lean into post-transcription intelligence rather than inline tagging: speaker diarization, sentiment analysis, entity detection, and summarization all sit on top of the transcript Whisper-AT / Interspeech 2023 Best Speech-to-Text Models (2026): 11 Options Best open source speech-to-text (STT) model in 2026. Universal-3 Pro posts a 3.1% AA-WER score, and the platform's reputation rests on the breadth of that intelligence layer rather than on tagging events inline during transcription itself Whisper-AT / Interspeech 2023 Best Speech-to-Text Models (2026): 11 Options Best open source speech-to-text (STT) model in 2026.

Google Cloud's Speech-to-Text, running Gemini 3 Flash, is 2.9% AA-WER, with language coverage as its standout trait Best Speech-to-Text Models (2026): 11 Options. Azure AI Speech's MAI-Transcribe-1.5 is 2.4% AA-WER, and it's noted for domain-specific custom models plus HIPAA-compliant medical transcription Best Speech-to-Text Models (2026): 11 Options. Amazon Transcribe is 4.1% AA-WER, solid on general transcription accuracy without being singled out as a leader on event tagging specifically Whisper-AT / Interspeech 2023 Best Speech-to-Text Models (2026): 11 Options.

These WER numbers come from different test sets and different scoring methods, so treat them as directional, not as a leaderboard you can rank on the third decimal place Best Speech-to-Text Models (2026): 11 Options. The Telnyx roundup behind several of these figures covers 11 models based on July 2026 benchmark data, and other sources cited are dated earlier in the year, so some numbers have already moved by the time anyone reads this Best Speech-to-Text Models (2026): 11 Options.

The open-source landscape for teams that need to self-host

Self-hosting buys control: data stays put, fine-tuning is unrestricted, and cost per hour drops at volume. It also hands the team operational responsibility for GPU capacity, updates, monitoring, and security.

NVIDIA's Canary-Qwen 2.5B currently tops the Hugging Face Open ASR Leaderboard at 5.63% average WER WESR Table 1. Its SALM architecture pairs a FastConformer encoder with a Qwen3-1.7B decoder, modified with LoRA adapters, trained on 234,000 hours of English speech, and it runs at 418x real-time (RTFx) WESR Table 1 Open ASR Leaderboard. It's English-only at 2.5B parameters, but it's one of the few open models that does double duty: straight transcription, or intelligent analysis including summarization and question-answering over the audio WESR Table 1 Open ASR Leaderboard.

IBM's Granite Speech 3.3 8B scores 5.85% WER on the same leaderboard, 8.18% on the separate Ionio clean-speech benchmark, and ships under Apache 2.0 WESR Table 1 Best open source speech-to-text (STT) model in 2026. It covers English ASR plus translation into several other languages, at roughly 8B parameters WESR Table 1 Best open source speech-to-text (STT) model in 2026. Independent testing found a 7.54% performance drop going from clean to noisy audio, a useful data point for anyone deploying it somewhere that isn't a recording studio WESR Table 1 Best open source speech-to-text (STT) model in 2026.

Whisper Large V3 remains the multilingual workhorse, covering 99-plus languages at 7.4% average WER with 1.55B parameters, and it's the base that Whisper-AT extends with tagging, at the cost of setting up that extension yourself Best open source speech-to-text (STT) model in 2026. NVIDIA's Parakeet TDT is the speed champion of the group: the 0.6B v2 variant hits an RTFx above 3,000, around 3,386 by Hugging Face's numbers, at 5.7% average WER, using a FastConformer-TDT architecture built for chunked streaming NVIDIA Parakeet / Hugging Face Open ASR Leaderboard Best Speech-to-Text Models (2026): 11 Options. It's English-focused and built for latency-critical pipelines, not for tagging depth NVIDIA Parakeet / Hugging Face Open ASR Leaderboard Best Speech-to-Text Models (2026): 11 Options.

Of this group, Canary-Qwen and Granite come closest to offering something beyond flat transcription. Canary-Qwen and IBM Granite are the closest to supporting intelligent analysis beyond pure transcription in the open-source tier, but dedicated non-speech audio tagging comparable to Whisper-AT still requires additional architecture work on top of these base models.

Pricing and use-case routing for developers

Voice agents run on a tight clock.

Watching that budget in production is its own discipline. TraceAI, an open-source tool built on OpenTelemetry, instruments the full STT, LLM, and TTS chain in a voice agent and catches latency creep or WER drift before a user notices something's off. That kind of observability matters more once non-speech tagging enters the pipeline, since it's one more processing step that can quietly eat into the latency budget without anyone noticing until users start complaining.

Pricing is the other variable nobody should assume their way through. Check the pricing page before the architecture diagram, not after.

Why non-speech tagging becomes load-bearing in conversational AI agents

In a voice agent, the STT output isn't the end product, it's the input to the LLM that decides what to say next. If laughter comes out as garbled text, or a stretch of background static gets hallucinated into a phrase nobody said, that corrupted context flows straight into the LLM's reasoning. The error doesn't stay contained to the transcript. It compounds, because the LLM now has to reason about something that never actually happened.

Non-verbal signals aren't decoration in a conversation, they're load-bearing. A pause can mean hesitation. Laughter can mean a request was a joke, not a literal instruction. A cough or a burst of background noise can mark where one person's turn ends and another's begins. An agent that can't tell these apart from garbage data will misroute the conversation or respond to something that was never actually said.

The scale here isn't small. Gartner's estimate that conversational AI will handle more than half of enterprise contact center volume by 2027 means this reliability gap, left unfixed at the STT layer, touches millions of real interactions rather than a handful of edge cases. Deloitte's 2026 Global AI Predictions report puts a number on how fast the deployment curve is moving: 25% of enterprises already using generative AI have deployed AI agents, and that figure is projected to roughly double by the end of 2027. At that scale, STT reliability is no longer just a debugging ticket but something a board asks about.

None of this works if it breaks the latency budget agents already run on tight margins for. Production agent architectures now combine streaming transcription, streaming LLMs, and parallel processing to reach the latency targets users expect, and introducing a non-speech tagging layer must not break this latency budget Whisper-AT / Interspeech 2023. Whisper-AT's under-1%-compute-cost result matters here as more than a footnote: it's the design precedent that proves tagging doesn't have to cost a system its speed Whisper-AT / Interspeech 2023.

Choosing a speech-to-text approach when non-speech audio tagging is a requirement

Four questions decide this WESR / ACL Anthology 2026. First, inline or post-transcription: does the product need event tags timestamped inside the transcript itself, or is a parallel report good enough? Second, granularity: will broad AudioSet-style categories cover the use case, or does it need something closer to WESR's 21-category vocal event taxonomy WESR / ACL Anthology 2026? Fourth, latency: real-time streaming or batch processing, since the answer changes what's tolerable in terms of added compute WESR / ACL Anthology 2026.

For teams that want intelligence breadth, sentiment, named entities, summarization, layered alongside basic event awareness, AssemblyAI's post-transcription intelligence stack is the documented option, though it works structurally differently from inline tagging. For teams self-hosting, Whisper Large V3 paired with the Whisper-AT extension remains the best-documented open-source route to non-speech tagging, while Canary-Qwen 2.5B delivers the strongest open-source transcription accuracy at 5.63% WER but needs extra architecture work bolted on before it tags anything Open ASR Leaderboard.

Which option fits depends on the same four questions: how inline the tags need to be, how fine-grained the taxonomy has to get, whether the deployment is hosted or self-managed, and how tight the latency budget actually is WESR / ACL Anthology 2026.

For anyone still comparing engines before committing to one, Telnyx's multi-engine benchmark roundup is a reasonable starting point for lining up the tradeoffs side by side. The honest takeaway, though, is that nothing on this list gets every axis right at once. Picking the two or three that matter most for the specific pipeline in front of you determines what to build around WESR / ACL Anthology 2026 WESR Table 1 Deepgram Aura-2.