Punctuation and Formatting in Automated Transcripts for Downstream NLP

A raw ASR transcript looks nothing like the sentences those downstream models trained on, and that gap, not model architecture, is what breaks entity recognition, translation, and summarization in production speech pipelines. Automated transcripts come out as a flat run of lowercase words: no periods, no commas, no capital letters, no difference between "january first twenty twenty-four" and "January 1, 2024." Every model sitting downstream of a transcript, translation, summarization, named entity recognition, sentiment scoring, trained on punctuated, cased, formatted text. Feed it a word stream instead and the model coasts through the same shallow pass. It's reading a format it has never seen before, and it won't manage the job.

Where punctuation, casing, and ITN live in the ASR pipeline

The typical ASR pipeline runs audio through a spectrogram, a neural acoustic model, and a decoder, sometimes paired with a language model to pick between similar-sounding word sequences. What comes out the other end still isn't a finished transcript. NVIDIA's ASR documentation describes a further stage: a punctuation and capitalization model, often BERT-based, bolted on after the words get decoded. Formatting sits outside the scope of transcription entirely. It's a separate job the raw output has to pass through before anything downstream can touch it.

That job usually gets framed as sequence labeling: every word gets tagged with a punctuation class, and the model has to make a call at every single word boundary, and that's a lot of small decisions adding up to one large one. Two designs compete here: cascade systems, where a standalone model like BERT gets bolted onto ASR output after the fact, and end-to-end systems that build punctuation into the transcription pass itself. Cascade is easier to swap out and debug. End-to-end is faster, since there's no second pass waiting on the first one to finish. Most production teams still reach for cascade, and that's the wrong default once latency matters at all: it's the safer choice for a research pipeline, not for anything shipping to users in real time.

Running alongside all of this is Inverse Text Normalization, or ITN, which turns spoken-form entities, dates, currencies, phone numbers, social security numbers, into their written equivalents. The name makes sense once the mirror image clicks into place: text-to-speech systems normalize "$25" into "twenty five dollars" so a machine can say it out loud. ITN runs that backward, turning "twenty five dollars" into "$25" so a machine can file it as data. NVIDIA's NeMo toolkit builds this as a two-stage weighted finite-state transducer pipeline. One stage classifies the semiotic tokens (is this a date, a currency, a measurement?), and the second turns them into written form. Skip this step, misconfigure it, or let it produce inconsistent output, and every system downstream inherits the mess without a word of warning. Nobody gets an error message. The garbage just flows through, quietly, and shows up three stages later as someone else's bug.

What NER loses when casing and punctuation disappear

Named entity recognition models trained on well-formatted benchmarks like CoNLL-2003 lose somewhere between 20 and 27 F1 points when handed raw ASR output, according to a multilingual NER benchmark. That's not a rounding error. That's the gap between a model a company can ship and one that sits in a demo forever.

Casing explains most of that collapse, and most teams have the priority backwards: they treat punctuation as the expensive fix and casing as a cosmetic afterthought, when the data says the opposite. AAAI 2020 research found that stripping capitalization alone drops NER performance by more than 40 F1 points on standard datasets. Casing is the bigger problem here, not the smaller one, and any team that spends its engineering budget fixing commas before fixing capitalization is solving the easy problem first. Capital letters aren't decoration. They're the strongest signal the model has for spotting a proper noun in the first place. Take "apple": lowercase, it's a fruit. Capitalized, it's a company with a market cap bigger than most national economies. Remove the capital and the model is guessing blind, because the one signal it trusted most just disappeared.

ITN failures compound the damage a different way. A phone number spoken as "eight hundred five five five one two one two" doesn't arrive as "800-555-1212." It arrives as eight separate word tokens, and an NER model trained to spot phone numbers as a single pattern instead sees nine unrelated words standing around with nothing connecting them. ACL 2024 research shows exactly where the damage concentrates: substituting ASR errors inside an entity span crushed NER F1 scores, while substituting errors outside entity spans dropped F1 by only 1.77 points. Errors inside a name or number are catastrophic. Errors everywhere else barely register. A 2024 study pairing Whisper with a large language model through a fine-tuned adapter, tested on the AISHELL-NER dataset, cut character error rate by 19.7% relative while also improving entity recognition, a fairly direct demonstration that fixing formatting fixes NER, not the other way around. Better NER on ASR output is a formatting problem wearing a model-architecture costume.

How missing punctuation degrades machine translation quality

Machine translation models expect cased, punctuated input, and ASR gives them neither, which is why researchers now insert a dedicated Punctuation Restoration Module, or PRM, between the two systems as standard plumbing rather than an optional extra. On the FLORES benchmark, unpunctuated ASR output caused a 20.7% relative drop in BLEU score, and the researchers who measured that gap built the PRM specifically to close it.

The IWSLT 2025 shared task produced a result that should worry anyone still treating word error rate as a quality gate: the Phi-4 ASR model posted the lowest WER in the competition and still produced worse translations than models with higher WER. Researchers traced the gap to inconsistent punctuation and casing, flaws WER can't see, since it only counts word substitutions, insertions, and deletions. A transcript with every word right and zero sentence boundaries scores perfectly on WER and falls apart the moment a translation model tries to find where one thought ends and the next begins. Without periods, the model either chops the input into fragments or runs everything together into one sentence with three unrelated ideas crammed inside it. Low-resource language pairs feel this acutely, because those models have less training data to lean on once the punctuation signal disappears. Anyone building translation on top of a quality gate that relies only on a single error-rate metric is measuring the wrong thing and calling it done.

Why punctuation is the enabling condition for meeting summarization at scale

Feed a long meeting transcript to an LLM in one pass and it hallucinates, even with the strongest models available. The standard fix chunks the transcript into topical sections first and summarizes each chunk separately. That chunking step depends entirely on sentence boundaries, and sentence boundaries come from punctuation and nowhere else. No periods, no natural place to cut the transcript, and the summarization system inherits a wall of text with no seams to work along.

A 2025 trends report found that more than 80% of businesses had already worked conversation intelligence tools into daily use over a year prior, which means transcript formatting problems stopped being a lab curiosity a while back. They're compounding across millions of recorded meetings right now, as this gets read. NVIDIA's Riva platform, built for meeting note-taking, makes the dependency explicit in its own architecture: streaming ASR refines punctuation and capitalization in real time specifically so downstream summarization has something usable to chew on.

The failure mode here goes past lower accuracy, into outright invention. Hand an LLM a wall of unpunctuated text and it doesn't fail gracefully. It builds structure that was never there, imposing boundaries and groupings that don't reflect what was actually said. Sentiment analysis hits a parallel wall: sentence-level polarity models need sentence boundaries to know what unit they're even scoring. Strip punctuation and clause-level sentiment collapses into flat, document-level static, telling a team a call was "mixed" when the first half went great and the second half fell apart.

ITN in detail: why spoken-form entities break structured data pipelines

ITN converts spoken numbers, dates, currencies, measurements, and phone numbers into the written form that databases, search indexes, and analytics tools expect. Skip it, and structured data lands as loose word tokens that nothing downstream can parse: not a search engine, not a spreadsheet, not a SQL query sitting there waiting for a WHERE clause that will never match.

The logic mirrors text-to-speech normalization in reverse. TTS turns "$25" into "twenty five dollars" so a machine can say it out loud; ITN turns it back so a machine can file it correctly. Production systems handle this three ways: rule-based weighted finite-state transducer grammars (Kestrel, Google's WFST-based system, hits 99.9% accuracy on Google's own TN test set, while other WFST grammar systems land somewhat lower), neural sequence-to-sequence models, and hybrids of the two. NVIDIA's NeMo toolkit runs its own two-stage WFST pipeline in production, classifying the semiotic type first and verbalizing into written form second.

Context is the genuinely hard case, and it's where rule-based systems start to strain. The same spoken phrase means different things in different domains: a string of digits after "patient is" is a dosage, not a dollar figure, and a medical transcript needs to resolve that differently than a financial one would. Recent research pushes toward end-to-end neural ITN that resolves that ambiguity through context instead of grammar rules, trading some interpretability and language coverage for sharper contextual judgment. When ITN fails on a date, a phone number, or a dollar figure, the workflow depending on that data doesn't just look sloppy. It doesn't start at all.

The streaming punctuation problem: latency versus accuracy in real-time pipelines

Streaming ASR has to punctuate each word before it knows what word comes next, a genuinely different problem than punctuating a finished transcript sitting quietly on disk. Waiting for more context improves accuracy. Waiting also adds latency, and a batch system never faces that trade-off, because it already holds the whole transcript in hand before it starts.

Without punctuation, a live stream has no clean sentence boundaries, and a translation engine or captioning system downstream can't segment or align text it can't parse. One production voice agent architecture solves this with a custom component, a PunctuatedBufferStreamer, that uses regex-based detection to catch punctuation in real time and slices the output into sentences before pushing them into a downstream queue. Punctuation here carries real weight. It's a runtime control signal deciding when a chunk of speech is ready to move to the next stage.

Latency budgets make the stakes concrete. Across a typical conversational AI stack, each stage, speech-to-text, language model inference, and text-to-speech, contributes its own slice of latency. Add it up and the delays stack on top of each other, and the total end-to-end latency adds up quickly even when each piece looks fast alone. Agents built for natural conversation need low-latency initial responses to feel human rather than robotic, and most production stacks struggle to meet that bar. NVIDIA's Nemotron 3.5 ASR Streaming model, a 0.6-billion-parameter system released in 2026, covers 40 language locales in one model with punctuation and capitalization built in alongside configurable low-latency modes, formatting handled inside the model instead of tacked on as a separate pass. Live captioning can absorb more delay than a voice agent before users notice. A voice agent can't get away with that. That gap in tolerance is what decides which punctuation architecture actually fits a given product, and picking the wrong one is why some voice agents feel like talking to a customer service line on hold.

Multilingual punctuation restoration: why the problem scales differently across languages

English has decades of annotated corpora and punctuation restoration models tuned to a fault. Most of the world's languages don't, and that includes the majority of the 70-plus languages that production voice platforms now claim to support, often on a slide with a checkmark next to each one and little else behind it. Data scarcity and linguistic complexity that English-centric models were never built to handle both stack up against low-resource languages at once, which is why "supports 70 languages" and "supports 70 languages well" are not the same claim, and vendors that let the two blur together are selling a checkmark instead of a working system.

Cadence, a model out of AI4Bharat and the Nilekani Centre, published at IJCNLP-AACL Findings 2025 in Mumbai, tackles this head-on. Built on Gemma3-1B, it covers all 22 scheduled Indian languages plus English, up from 14 languages in the prior state of the art. It handles 30 distinct punctuation classes, not the small set of punctuation classes that English-only systems typically handle, including Indic-specific marks like the Devanagari danda and punctuation combinations that simply don't exist in English. Its training data draws from Sangraha-verified sources, IndicVoices, a translated version of Cosmopedia, and IndicCorp-v2, curated deliberately to keep low-resource languages represented instead of letting the largest languages dominate the mix by default, which is what tends to happen when nobody curates on purpose.

Architecturally, Cadence converts Gemma3-1B into a bidirectional model using a Masked Next Token Prediction objective, which allows efficient non-autoregressive sequence tagging, a choice aimed at the throughput demands of a live pipeline rather than a leaderboard score. Even so, the model's own analysis flags two persistent problems: domain shift, where a model trained on clean written text struggles against spontaneous speech full of false starts and backtracking, and rare punctuation marks that show up too infrequently in training data to classify with confidence. The downstream stakes run higher here too. Low-resource language pairs in machine translation degrade faster when punctuation goes missing, precisely because those models have the least redundancy to fall back on once the signal disappears.

What WER does not measure and why formatting metrics matter separately

Word Error Rate counts substitutions, insertions, and deletions at the word level. It says nothing about punctuation, nothing about casing, nothing about whether ITN rendered a date correctly. All three can fail completely and WER won't budge, which makes WER a bad gatekeeper for anything but the narrowest transcription task.

The IWSLT 2025 result involving Phi-4 is the cleanest demonstration available: lowest WER in the competition, worse translation output than competitors with higher WER, because WER graded a transcript as correct that was structurally broken in every way that mattered to the model reading it next. A transcript can be word-perfect and still unusable: every word transcribed correctly, zero sentence boundaries, no proper nouns capitalized, every phone number and date rendered as spoken words instead of structured data waiting to be queried.

Teams that gate transcript quality on WER alone have no way of watching their formatting pipeline degrade, because the degradation stays invisible on the one dial they're checking. Punctuation F1, casing accuracy, and ITN accuracy measured against a properly formatted reference belong next to WER as separate scores, not folded into it. A speech-to-text API that reports WER and says nothing about formatting quality is handing developers half the picture and calling it the whole thing.

What well-formatted transcripts enable that raw transcripts cannot

Sentiment analysis, entity recognition, translation, summarization: every one of these tasks assumes the sentence structure that punctuation, casing, and ITN provide. Strip that structure out and the results aren't just weaker. They're unreliable in a way baked into the setup before the model ever runs a single token.

The dependency chain runs one direction only. Punctuation makes sentence splitting possible. Sentence splitting makes topical chunking possible. Topical chunking is what keeps a summarization model from hallucinating its way through an hour of meeting audio. Pull the first link out and the rest of the chain never gets the chance to matter, no matter how good the model sitting at the end of it happens to be.