Speaker Diarization in Multi-Party Meeting Transcription

Speaker diarization answers one question: who said that? It sounds trivial until a transcript from an actual conference room lands on someone's desk with six voices tangled into one paragraph. Automatic speech recognition turns sound into words; diarization decides which mouth those words came out of. Skip that step and a meeting transcript is just a wall of text nobody can act on. The market for AI meeting tools keeps growing precisely because the "who spoke when" problem underneath it remains unsolved, not despite that fact.

The accuracy gap that makes diarization its own problem

Speech recognition had one of the best three decades in applied AI. Word error rates dropped from around 40% to under 3%, a curve so steep it's practically the industry's favorite slide. Diarization refused the invitation. Diarization Error Rate for spontaneous, multi-party conversation still sits in the 10 to 20% range across leading academic and commercial systems, and that gap explains why this field moves so much slower than ASR did.

This isn't a laziness problem. It's a structural one. ASR is a sequence-labeling task: audio comes in, words come out, in order. Diarization has to track identity continuously through time without knowing in advance how many people are even in the room, while several of them talk over each other. Most clustering-based diarizers assume every audio frame belongs to exactly one speaker, and meetings break that assumption constantly. Because the pipeline runs in stages, a diarization mistake doesn't just mislabel a speaker, it files that person's words under someone else's name entirely. A DER of 10 to 20% translates into a real chunk of every meeting ending up misattributed, enough that anyone using the transcript for something that matters still needs a person to fix it by hand afterward. The real cost of this technology was never the error rate. It's the cleanup that error rate guarantees.

How a diarization pipeline is structured and where each stage can fail

Picture an assembly line with four stations, and any one of them can hand the next station garbage. Voice Activity Detection goes first, deciding what counts as speech versus silence or background hum. Get that wrong and everything downstream inherits the mistake: pass HVAC noise off as speech, or drop a quiet aside, and the rest of the pipeline builds on a bad foundation.

Next comes speaker embedding extraction, where a neural network compresses a chunk of audio into a fixed-size vector meant to capture vocal identity: tone, pitch, cadence, the acoustic fingerprint of a person's voice. Clustering takes those vectors and groups them into speaker buckets. It's the standard approach in commercial systems, and it's structurally blind to overlap. If two people talk at once, that frame's embedding is a blend of two voices, and clustering has no native way to split it back apart.

Overlap detection gets bolted on as a patch: a separate module flags the frames where two people spoke at once so the system can treat them differently. But overlap detection is a hard problem on its own, not a clean fix for the harder problem underneath it. Once a cascaded system sends the wrong audio slice to the ASR module, nothing downstream can undo it. Add far-field, multi-microphone recordings into the mix, and the front end also needs dereverberation and sometimes direction-of-arrival estimation before any of this even starts. Clustering was built for a world of clean two-person phone calls. Running a conference room through it means asking it to do a job it was never designed for, and that mismatch is the recurring failure in how vendors market this technology: they benchmark on the easy case and ship into the hard one.

The specific conditions that break even well-designed systems

Two-speaker phone calls, the dataset most early diarization research trained on, are a different animal from a conference room. Overlapping speech shows up far more often in multi-party meetings than in any two-person call, and systems that need to be told the speaker count in advance can't handle a meeting where that number isn't known until people actually show up.

Distance from the microphone matters more than most people assume. The M2MeT dataset captured meetings at distances of 0.3 to 5.0 meters, a range wide enough to degrade embedding quality on its own, and reverberation off conference room walls compounds it further.

Voices also drift within a single session. Someone leans back, someone else leans toward the microphone to make a point, energy rises and falls over a ninety-minute meeting, and the same person's embeddings can shift enough to split into two separate clusters, as if they'd been swapped for a stranger halfway through. Code-switching, where a participant shifts languages mid-sentence, throws off systems trained on monolingual speech. And when two speakers sit close together, the spatial cues that would normally help tell them apart collapse. The CUHK-Tencent team's M2MeT submission trained data augmentation specifically to handle small angular differences between speakers sitting near one another, which says a lot about how narrow the margin gets once bodies are packed around one table.

What the M2MeT challenge revealed about the state of the art in real meeting conditions

M2MeT, run at ICASSP 2022, is the closest thing this field has to a stress test built on purpose. The dataset: 120 hours of real Mandarin meeting speech, hand-annotated, recorded on an 8-channel far-field microphone array alongside individual near-field headsets. Two tracks ran in parallel, one for diarization and one for transcribing overlapping speech directly.

ByteDance's VolcSpeech system achieved a DER of 5.79% on the evaluation set, among the top results on the diarization track, and nobody got there with one clever model. It took multi-channel signal combination, dereverberation, direction-of-arrival-assisted overlap detection, two different embedding architectures (ResNet-101 and ECAPA-TDNN), and a modified DOVER-Lap fusion step stitching multiple sub-systems together. That's the honest lesson from M2MeT: meeting diarization gets solved by stacking specialized components, not by training one bigger model. Anyone selling a single-model fix for meeting audio is selling something that hasn't been stress-tested. CUHK-Tencent's second-place system made a similar admission in a different form, employing a target-speaker voice activity detection framework, effectively conceding that acoustic embeddings alone don't carry enough information. Follow-on benchmarks (M2MeT 2.0 at ASRU 2023, Microsoft's NOTSOFAR-1 in 2024, the multi-modal M3SD dataset in 2025) have all made the test harder rather than easier. The frontier hasn't stopped moving, and nobody serious expects it to settle soon.

How modern systems handle the speaker-count and overlap problems differently

End-to-End Neural Diarization throws out the cluster-then-detect pipeline entirely and trains one network to detect speech activity for every speaker at once, handling overlap natively instead of patching it in after the fact. Attractor-based versions of EEND go further, letting the network handle however many speakers show up in a given session rather than forcing a fixed count going in, which matters a great deal when a meeting might have four people or fourteen.

Target-speaker voice activity detection flips the framing again: feed the model a speaker's embedding as a reference, and it estimates, frame by frame, whether that specific person is talking, run in parallel for everyone in the room. Real-time systems add another constraint on top of all this: they can't peek too far into the future. NVIDIA's Sortformer caps its right-side attention window at 7 frames, roughly 560 milliseconds, applied with partial probability during training, specifically so the model doesn't learn to lean on audio that hasn't arrived yet in a live setting.

On the open-source side, pyannote.audio reports roughly 10% DER on standard benchmarks and quietly sits underneath several commercial diarization APIs most people have never heard of by name. A newer entrant, Diarizen, released in 2025, keeps pyannote's basic architecture but swaps in WavLM representations for richer acoustic features, and beats pyannote across multiple test sets doing exactly that: same scaffolding, better inputs. That's the pattern worth betting on, and it cuts against the industry's own hype cycle. Architecture is getting commoditized. The wins are coming from better representations feeding old designs, not from newer designs chasing novelty for its own sake.

Where LLMs are entering the diarization pipeline

The cascaded pipeline has a ceiling built into its own architecture: once diarization makes a call, ASR runs on whatever audio slice it was handed, and nothing downstream can use the meaning of the words to go back and question that call. Google's DiarizationLM, presented at Interspeech 2024, works around this without rebuilding the pipeline. It applies an LLM after transcription runs, using language context to clean up speaker labels.

Other approaches remove the seam altogether instead of patching around it. The DM-ASR method integrates diarization information directly into an LLM's decoding process, so the model produces transcript text and speaker labels together instead of in sequence. SpeakerLM, also 2025, claims to be the first audio-text multimodal LLM built to handle full speaker diarization and recognition inside one architecture, rather than as two systems handed off to each other. MT-LLM, presented at ICASSP 2025, trains a language model to transcribe multiple talkers under flexible instructions, another attempt at collapsing diarization and transcription into a single act. Recent multimodal diarization work pushes past audio entirely, combining visual cues alongside speaker embeddings and transcript context. A 2026 system called TagSpeech (arXiv:2601.06896) aims for fine-grained temporal grounding in one end-to-end model. The direction across all of these is the same: stop treating "who spoke" and "what was said" as two separate questions asked by two separate systems. The seam between them is exactly where errors used to hide.

How commercial APIs implement diarization today and what the tradeoffs look like

No vendor wins on every axis, so the choice comes down to which tradeoff matters for the use case at hand. Picking on accuracy alone is how teams end up locked into a vendor that can't meet their compliance requirements six months later. Several AI audio platforms have built speech-to-text APIs that surface diarization as a feature; ElevenLabs, for instance, offers speaker recognition through its Scribe model. AssemblyAI's Universal-3 Pro handles diarization across 95 languages works reasonably well on noisy conference room audio, and carries SOC2 Type 2 certification for enterprise buyers who need that paperwork. Speechmatics' Flow platform lets a customer set a maximum speaker count anywhere from 2 to 20, covers over 30 languages, and offers on-premise deployment alongside cloud, the deciding factor for any organization whose data residency rules take cloud-only vendors off the table entirely.

Gladia pairs Whisper-derived Solaria transcription with pyannote for diarization, and offers an enhanced diarization tier aimed at edge cases and difficult audio, a sensible pick for teams already built around the Whisper stack who just need speaker labels bolted on. Google Cloud's Chirp 3 covers more than 85 languages with automatic punctuation and word-level confidence scores, and its latest update adds Arabic regional variants and diarization support for select locales.

Then there's the efficiency angle, which gets less attention than accuracy but matters just as much outside the data center. Picovoice's Falcon, benchmarked as open source, reaches accuracy comparable to pyannote while using 221 times less compute and 15 times less memory (0.1 GiB versus 1.5 GiB). Nobody wants to stream a board meeting to a third-party server just to find out who was talking, and that gap is the entire argument for running diarization on-device instead. For teams whose diarization needs live entirely inside an existing videoconferencing contract, the built-in tools are already doing real work: Zoom reports 99.05% transcription accuracy against Webex's 98.71%, with Zoom's transcripts logging 27% fewer errors. None of this adds up to one obvious winner. Language coverage, accuracy under overlapping speech, real-time versus batch processing, on-premise availability, and compliance certifications all pull in different directions, and the right vendor depends entirely on which of those a given team can't live without.

What accurate speaker diarization actually enables downstream

Once every utterance carries a speaker label and a timestamp, a transcript stops being a document and starts being data. It can be queried, summarized, and handed off to something else automatically. A summarization model that knows who said what can assign an action item to a named person instead of producing a to-do list nobody owns. Speaker-labeled segments also feed directly into training data for other conversational AI systems, and diarization quality there shapes what those downstream models learn about how real conversation actually works: who interrupts whom, how turns get taken.

The infrastructure spend behind meeting AI has already happened; most enterprises have some version of it running. What decides whether that spend pays off is whether the diarization underneath it is accurate enough to trust, not whether the transcript looks polished on screen.

In medicine, law, and governance, that trust threshold isn't optional. A deposition transcript, a board meeting record, a regulatory hearing: all of them need speaker attribution solid enough to survive being challenged later, and a 10 to 20% DER is the kind of gap that can surface as a serious problem once attribution is tested under pressure. Real-time conversational agents raise the stakes further, since an agent joining a multi-party call has to know who's talking in a given turn just to respond to the right person. That turns diarization from a transcription feature into something the system depends on to function at all. Every architectural idea covered here (end-to-end neural diarization, target-speaker detection, LLM-based correction, multimodal fusion) is aimed at the same target: a record of who said what that's accurate enough for someone to actually rely on.