Neural Codec Language Models as a Speech Generation Paradigm

For most of TTS history, the core problem was that audio is continuous and language models are not. You can't feed a waveform to a transformer the way you feed words. The codec paradigm solved this with a surprisingly elegant sleight of hand: compress audio into discrete integer codes first, then treat those codes like tokens.

The compression method that made this tractable is Residual Vector Quantization. An encoder maps a raw waveform into a hierarchy of discrete codebooks, where each successive codebook captures the residual detail the previous one missed. The first codebook carries the coarse structure: speaker identity, prosody, the broad shape of the utterance. Subsequent codebooks fill in progressively finer acoustic detail. A decoder then reconstructs a waveform from those codes. The reconstruction is lossy, but perceptually, at reasonable bitrates, it sounds clean.

Two codecs anchored the early research field: SoundStream, from Google in 2021, and EnCodec, from Meta in 2022. They became the shared substrate for nearly all foundational codec-LM work, functioning less like proprietary systems and more like reference implementations that let different research groups build on comparable footing. Then things got crowded fast. SpeechTokenizer, DAC, FunCodec, RepCodec, Vocos, and a 2024 to 2025 wave including ComplexDec, FreeCodec, and UniCodec each optimized for different tradeoffs: domain robustness, streaming latency, disentanglement of speaker from content, low-bitrate operation. The Codec-SUPERB benchmark emerged to evaluate these systems systematically once shared evaluation standards became necessary.

Once audio is a sequence of discrete integer codes, the modality barrier between speech and text effectively dissolves. The same transformer machinery that predicts the next word can predict the next audio token, and every generalization property that language modeling had spent a decade accumulating was now, in principle, available to speech. The implications of that are significant enough that the field is still working through them.

AudioLM and VALL-E: the two papers that defined the paradigm

Two papers, published within months of each other, established what codec-LM-era speech synthesis would look like. If you only read two papers to understand where TTS is today, read these.

AudioLM, from Google in 2022, was the conceptual breakthrough. It framed speech generation as hierarchical token-sequence prediction without any text input: semantic tokens generated first to capture linguistic content, then acoustic tokens conditioned on those semantic tokens to fill in voice and detail. The output demonstrated continuity and naturalness that prior systems couldn't approach, and it did so without hand-crafted features or manually engineered intermediate representations. The pipeline was gone. The model learned what the pipeline used to compute.

VALL-E, from Microsoft in January 2023, pivoted the same core idea toward zero-shot TTS and produced what was, by industry standards, a shock result. Given a three-second enrollment clip from an unseen speaker, the model could generate arbitrary speech in that speaker's voice, matching not just timbre but emotional tone and speaking style. The architecture was hybrid: an autoregressive model generated the first RVQ codebook, capturing coarse prosody and speaker identity, while a non-autoregressive model filled in the remaining codebooks in parallel.

Three seconds. Not thirty minutes of studio recordings, not a fine-tuning run, not a dedicated speaker model. The system had internalized speaker space from pretraining on tens of thousands of hours of audio. Microsoft was notably candid about the implications: they withheld public weights, cited misuse risk explicitly, and committed to developing detection methods alongside the model. That caution was not paranoia. It was an accurate reading of what the model could do, and the field spent years catching up to that honesty.

Together, AudioLM and VALL-E established the paradigm's core claim: treat speech like language, and you inherit language modeling's capacity to generalize.

The autoregressive vs. non-autoregressive tradeoff that every subsequent model had to solve

Every codec-LM paper published after VALL-E had to reckon with the same tension. Autoregressive generation, predicting each token conditioned on all prior tokens, produces coherent, naturalistic output. It also requires sequential decoding, which is slow, and errors compound over long sequences in ways that become audible. Non-autoregressive generation produces tokens in parallel, which is faster, but it sacrifices the fine-grained temporal dependencies that make speech sound connected rather than assembled.

VALL-E's hybrid was a pragmatic compromise: autoregressive for the first codebook, non-autoregressive for the rest. It worked well enough to define the field's baseline, but inference cost remained high enough to strain real-time use cases.

SoundStorm, from Google in 2023, attacked the speed problem directly. Rather than flattening multi-level codec tokens into a single sequence and running AR decoding over it, SoundStorm modeled those tokens in parallel structures, achieving a speedup of roughly two orders of magnitude over autoregressive approaches while preserving perceptual quality. What SoundStorm actually demonstrated is that the speed-quality tradeoff wasn't a fixed physical constraint. It was an architectural choice someone hadn't made yet.

VALL-E 2, published by Microsoft in June 2024, approached it differently. Rather than redesigning the generation structure entirely, it addressed the fragility of long autoregressive sequences through two targeted interventions: Grouped Code Modeling, which organizes codec codes into groups to shorten the effective sequence length the AR decoder must traverse, and Repetition Aware Sampling, which tracks token repetition history during decoding to prevent the looping artifacts that plagued earlier models at longer outputs. Both interventions target production-length generation, not the short-utterance demonstrations that dominated early codec-LM research. Solving the tradeoff required changes to the decoding algorithm, not just the model architecture. That distinction separates systems that work in demos from systems that work in production.

What human parity on benchmark evaluations actually means — and what it leaves open

VALL-E 2, in June 2024, reported human parity on LibriSpeech and VCTK: listeners could not reliably distinguish synthesized speech from human recordings on naturalness and speaker similarity. That result is real. It also warrants scrutiny before you put it on a slide deck.

LibriSpeech and VCTK consist predominantly of read speech from cooperative speakers in controlled acoustic conditions. They are not wild-caught audio. They lack spontaneous disfluencies, emotional volatility, cross-speaker conversational dynamics, reverberant environments, or low-resource languages. Human parity on those benchmarks means human parity on a narrow and somewhat idealized slice of real-world speech, not on synthesizing a convincing angry customer service call recorded over a spotty mobile connection.

What the benchmarks don't capture is a longer list. Fine-grained expressive conditioning, specifically the ability to modulate a single attribute like urgency or emotional intensity at the clause level rather than the utterance level, remains an open problem the research literature consistently flags. High-level style tokens have improved coarse-grained style transfer substantially, but they don't reach the granularity that voice directors or audiobook producers actually need. Long-form generation continues to strain codec LMs in ways short-utterance benchmarks don't reveal. Robustness in noisy or multilingual conditions still varies meaningfully across systems.

Human parity on controlled evaluations is a genuine threshold. The remaining frontier is narrower than it was. But narrower is not the same as closed, and confusing the two leads to deployment decisions that fail in exactly the conditions the benchmarks left untested.

How the ecosystem extended beyond TTS: multi-task, multilingual, and cross-modal codec LMs

Once it became clear that codec token representations generalized, the field branched quickly from TTS into adjacent problems. The codec representation gets reused; the conditioning signal and task head change.

SpeechX, published in IEEE/ACM Transactions on Audio, Speech, and Language Processing, unified codec language modeling across zero-shot TTS, noise suppression, and speech editing through task-dependent prompting. Same model, different conditioning. VioLA took the next logical step, building a single codec LM that handled recognition, synthesis, and translation together, collapsing speech understanding and generation into one architecture. The appeal of that consolidation is obvious to anyone who has managed a multi-model speech pipeline in production.

Cross-lingual and multilingual extensions emerged in parallel. VALL-E X applied the codec-LM framework to cross-lingual voice synthesis: generate a target-language utterance in the source speaker's voice, even if that speaker never spoke the target language during training. PolyVoice applied a similar approach specifically to speech-to-speech translation. These systems represent a direct architectural response to multilingual market demand, which becomes relevant in the product section below.

On scaling, UniAudio and BASE TTS explored whether the scaling laws observed in text language models transfer to audio, pushing codec LMs toward a billion parameters trained on hundreds of thousands of hours of data. The answer is yes, with meaningful quality gains at scale, though the gains are not perfectly linear and the compute costs are not trivial.

ELLA-V and RALL-E addressed a persistent instability problem in VALL-E-class models: the tendency to produce phoneme errors or unstable outputs on difficult text inputs. Both added explicit speech-text alignment prediction to the decoding process, improving robustness without abandoning the core architecture. These are the kinds of papers that don't generate press releases but that production engineers quietly depend on.

Where the current research frontier is actively moving

The research frontier in 2025 is less about proving the paradigm works and more about making it work in conditions the original papers didn't address.

Streaming and low-latency architectures are the most commercially urgent thread. TS3-Codec and related transformer-based streaming codecs are designed to support real-time generation without buffering, a prerequisite for voice agents. The gap between a system that produces high-fidelity audio and a system that produces adequate audio in under 500 milliseconds does not close on its own; it requires dedicated architectural work, and a meaningful number of vendors have yet to do it.

Disentanglement is the most conceptually interesting thread. FreeCodec and related work attempt to separate speaker identity, prosody, and linguistic content into distinct token streams rather than entangling them in a shared codebook. If that separation holds at generation time, it enables fine-grained attribute control that current conditioning approaches can only approximate. Research results are promising; production deployments of this approach remain limited.

Hierarchical codec architectures for long-form audio are addressing a limitation that became visible as the field moved beyond short-utterance demonstrations. PyramidCodec, developed primarily in the music domain, applies a hierarchical approach to long-form content; the same principle applies directly to audiobook-length speech generation, where flat RVQ representations begin to struggle with coherence.

Safety and detection research has become a co-equal priority alongside generation quality. As zero-shot voice cloning from short clips becomes standard, detection and watermarking are structural requirements. The research community is treating them as such. Practitioners should too, and the ones who aren't are storing up a problem for later.

Codec-SUPERB's push toward evaluation standardization signals something real: the field is trying to become an engineering discipline rather than a collection of compelling demos. That's progress worth acknowledging.

What the paradigm shift produced in the product and market layer

The market data reflects what the technology actually delivered. Neural and AI voices held approximately 67% of text-to-speech market revenue in 2025, according to Mordor Intelligence. The broader AI voice generator market was estimated at around USD 4.16 billion in 2025, with projections reaching USD 20.71 billion by 2031 at a 30.7% compound annual growth rate, per MarketsandMarkets. Growth at that rate doesn't happen from incremental improvements. It happens when quality crosses a threshold buyers can perceive without being told to look for it.

The fastest-growing subsegment, at a 34.7% CAGR from 2025 to 2031, is the developer API and SDK layer. The codec paradigm's biggest near-term commercial expression is programmatic access, not consumer products. Builders are embedding this capability into their own products, which means the quality of the underlying model compounds across a large surface area of applications the underlying vendor never directly touches.

Customer service and IVR applications held the largest application share in 2025, at a plurality, per Mordor Intelligence. These are the deployments where low-latency, naturalistic voice matters most commercially and where the quality gap between parametric TTS and codec-LM-derived systems is most audible to end users.

ElevenLabs, identified alongside Microsoft and NVIDIA by MarketsandMarkets as a leading player in 2025, reached a $3.3 billion valuation. The valuation reflects both commercial traction and the structural fact that a platform spanning content creation and real-time agent deployment, built on continuously updated production models, is a different kind of asset than a single-use voice API.

Multilingual demand is visibly fragmenting the English-dominant market structure. English held around 52% of market share in 2025, but Hindi alone was growing at a 13.42% CAGR. Sarvam AI's Bulbul-v2, covering 11 Indian languages as of May 2025, represents at the product layer the same pressure that VALL-E X and PolyVoice represent at the research layer. Whether the systems are responding fast enough is a separate question nobody has a clean answer to yet.

How builders and enterprises should think about deploying codec-LM-era voice today

The quality bar has shifted. Evaluating codec-LM-derived systems on word error rate alone, the legacy metric for TTS quality, misses the point. The relevant dimensions are speaker similarity, prosody naturalness, zero-shot generalization to novel speakers, and robustness across speaking styles. Build your evaluation rubric around those dimensions, not the metrics your team inherited from 2018.

Latency and fidelity are separate dimensions, and conflating them produces procurement decisions that don't match the actual use case. Streaming architectures capable of delivering audio in under 500 milliseconds suit real-time voice agents. Higher-latency, higher-fidelity pipelines suit audiobook production, commercial voiceover, and dubbing workflows where generation latency is irrelevant. Know which problem you're solving before you evaluate vendors.

Voice cloning tier selection is consequential and frequently underspecified in RFPs. Instant cloning from one to five minutes of audio suits rapid prototyping and use cases where voice consistency matters more than acoustic perfection. Professional cloning requiring thirty or more minutes of high-quality recorded audio is the correct path for broadcast, audiobook, and commercial voiceover work. Evaluating the technology at a lower tier than your production use case requires, then drawing conclusions about the ceiling, is a methodological error that wastes everyone's time.

ElevenLabs offers a production deployment spanning content-creation workflows and real-time conversational agents on the same underlying models, with a developer API offering Python and Node.js SDKs and WebSocket support. Its Twilio ConversationRelay integration illustrates what codec-LM-era voice looks like deployed at CPaaS scale.

Multilingual coverage is a first-order architectural question, not a feature added after the fact. Codec LMs trained natively on diverse language data outperform systems that bolt on language support post-hoc. In procurement decisions, probe for this directly: ask how the model was trained on non-English data, not just which languages appear on the feature page.

Responsible deployment is not a compliance checkbox. Microsoft's caution at VALL-E's release in 2023 was prescient and largely underappreciated at the time. As zero-shot voice cloning from short clips becomes standard practice, consent frameworks, detection capabilities, and watermarking are prerequisites for sustainable deployment. Retrofitting them is considerably harder than building them in from the start.

Open-source alternatives, including Coqui XTTS, Bark, and locally deployable open-weight models, remain viable for teams that require full model ownership, GPU infrastructure control, or air-gapped environments. The tradeoffs are real: maintenance burden, a quality gap relative to continuously updated production APIs, and the engineering overhead of running your own serving infrastructure. Those are defensible choices for specific contexts. They are not defaults that carry inherent virtue simply because open-source sounds principled.