TTS Acoustic Architectures From Tacotron 2 to VITS

Before neural networks took over, TTS ran on two competing methods, and both had a distinct way of sounding wrong. Concatenative synthesis stitched together fragments of recorded speech pulled from a large database, and it sounded uneven at the joins, like a sentence assembled from index cards someone shuffled and dealt back out of order. Statistical parametric synthesis (SPSS) generated speech from statistical models of vocal features instead of recordings, and it came out smoother but robotic-sounding, the way a photocopy of a photocopy keeps the outline and loses the detail.

Neither method solved the underlying problem, so early neural TTS just inherited the structure both had been built on, consisting of text analysis, an acoustic model, and a vocoder, trained separately and bolted together only at inference. The logic behind the split made sense at the time. Predicting acoustic features from text and turning those features into a waveform are different computational problems with different failure modes, and keeping them separate meant a vocoder could be swapped or upgraded without retraining the whole system from scratch.

That separation bought modularity. An engineer could isolate which block was misbehaving without tearing apart the rest. It also baked in two costs that took years to unwind: errors from one stage bled downstream with no way to correct them, and generation ran strictly sequentially, block after block, with no shortcut for speed. Nobody sat down and chose that tradeoff. It's just what happens when you build a pipeline before you know how to build one machine.

Tacotron 2's achievements and the limits of its autoregressive design

Tacotron 2, released by a major technology company, made that two-stage structure sound genuinely human in a way earlier systems had not. Its acoustic model was a recurrent sequence-to-sequence network that turned character embeddings into 80-channel mel-spectrograms, generated in 50-millisecond frames with a 12.5-millisecond hop, and those spectrograms were converted to audio by a modified WaveNet vocoder. Two components, trained separately, joined only at inference: the same handoff structure that had defined TTS since the concatenative era, just with better parts.

The results closed a gap that had looked unbridgeable. Tacotron 2 scored a mean opinion score (MOS) of 4.53, against 4.58 for professionally recorded human speech reading the same scripts. That's a rounding error in perceptual terms, and it marked the point where neural TTS stopped being a research curiosity and became a legitimate stand-in for recorded voice.

The quality came at a cost, and the cost has a name: attention collapse. Tacotron 2's encoder and its autoregressive decoder learned the alignment between text and audio implicitly, through attention, rather than being told outright which phoneme lines up with which frame. Attention is elegant when it works and unpredictable when it doesn't. Certain input sequences could cause the alignment to fail, producing unreliable audio with no clean fix built into the architecture. Worse, because the decoder generated one mel-spectrogram frame at a time conditioned on the last, none of it could run in parallel during inference. Making sequential decoding fast enough for practical deployment required substantial engineering effort. Nobody advertises this trade: the more naturally a model listens to itself while it talks, the harder it is to make it talk fast.

The vocoder half of the pipeline and the race to make waveform generation fast enough to matter

The vocoder side had its own version of this problem, and a separate track solved it. WaveNet, the model Tacotron 2 leaned on, produced audio that matched or beat human raters' judgment of quality at a level earlier vocoders had not reached. It also generated one audio sample at a time, autoregressively, at a computational cost that made it a research benchmark rather than something anyone could ship.

What followed was a straightforward speed chase, the audio equivalent of an arms race fought with flow-based math instead of missiles. WaveGlow, Parallel WaveGAN, and HiFi-GAN all set out to hit WaveNet's quality bar without its inference cost, using flow-based and adversarial training methods that generate audio in parallel instead of sample by sample. HiFi-GAN in particular became widely adopted in TTS stacks, the kind of component that earns its place by staying out of the way.

A different branch of that race used LPCNet, a lightweight neural vocoder built to run faster than real time on a single CPU core, no dedicated graphics processor required, with IBM Research demonstrating its integration into a full sequence-to-sequence TTS pipeline. Sequence-to-sequence work on LPCNet showed it could match or exceed WaveNet-backed systems on MOS degradation, with quality loss of roughly 6.1 to 6.5 percent and expressiveness loss of roughly 6.3 to 7.0 percent relative to raw PCM audio, while running in real time on ordinary hardware. By the time VITS entered the picture, the vocoder half of the problem was mostly solved. What remained unsolved was the seam itself: an acoustic model and a vocoder trained apart, asked to cooperate at inference despite never once seeing each other during training.

FastSpeech and the non-autoregressive turn: trading attention for explicit duration

FastSpeech went after the fragile, sequential attention mechanism directly. Instead of an autoregressive decoder guessing alignment through attention, FastSpeech used a feed-forward Transformer paired with an explicit duration predictor, a component that tells the model directly how many audio frames each phoneme should occupy. No implicit alignment, no attention collapse, no guessing games at inference time.

The speed gain wasn't subtle. Because every output frame could be generated at once instead of one after another, FastSpeech produced mel-spectrograms substantially faster than autoregressive systems like Tacotron 2. Waiting for a kettle to boil takes time. Having the water already hot does not.

FastSpeech 2 pushed the idea further, adding pitch and energy as explicit inputs the model conditions on rather than qualities it has to infer on its own. That made prosody controllable in a direct, dial-like way, but it costs something real. Speech has a one-to-many relationship with text: the same sentence can be spoken a dozen different ways, and a model predicting one deterministic duration and pitch contour per phoneme flattens all of that into a single version, every time. FastPitch extended the same logic by predicting pitch contours at inference for more expressive output, while keeping the parallel, explicit-duration structure that made the whole family fast. Explicit control bought speed and predictability. It just spent naturalness to pay for it, and that bill comes due on every single output.

VITS collapsing the pipeline into a single jointly trained model

VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech) is where the two-stage structure finally broke apart for good, and it should have happened years sooner. Instead of a separately trained acoustic model handing off to a separately trained vocoder, VITS trains a prior encoder, a posterior encoder, and a decoder jointly, inside a normalizing flow framework, with no external vocoder bolted on afterward. Text goes in, a waveform comes out, and every piece of the system gets optimized together toward that one output.

The alignment problem that plagued Tacotron 2 gets solved differently here. VITS uses Monotonic Alignment Search (MAS) to learn text-to-audio alignment explicitly during training, and that alignment supplies the phoneme durations used to supervise a stochastic duration predictor. The result is stable and interpretable in a way attention never was, and it kills off the specific failure mode where certain sentences would come out garbled with zero warning.

The word "stochastic" in that duration predictor's name is doing real work. FastSpeech's duration prediction is deterministic: same text in, same timing out, every time. VITS samples duration instead, so the same sentence can render with different rhythm and pitch across multiple generations, preserving the one-to-many variability that FastSpeech's precision had traded away. Layered on top is an adversarial discriminator, borrowed straight from GAN training, that pushes the system toward waveforms that sound convincing to a listener rather than merely close to a spectrogram target on paper. Every part of the model gets judged by the same yardstick: does the final audio sound right, not does the intermediate representation look right on a graph. That's the correct standard, and most of the field took years to catch up to it.

VITS becoming a platform rather than just a model: multilingual extensions and zero-shot voice cloning

VITS turned out to be more useful as a foundation than as a finished product, which is the highest compliment an architecture can get. YourTTS extended it toward zero-shot multi-speaker synthesis and voice conversion: the same trained model could generate a new, unseen speaker's voice from a short audio prompt, no dedicated training run per speaker required.

A large-scale multilingual speech project from another technology company pushed the same VITS-based architecture to cover more than 1,100 languages, the largest language-coverage demonstration built on this framework by a wide margin. That scale came with a visible cost: coverage at that scale comes with quality trade-offs, and that gap exposes the real tension whenever one architecture gets stretched across too many targets at once. Breadth and depth pull against each other. Pick a lane, or pay for the lane you didn't pick.

What let VITS absorb these extensions so easily traces back to the joint training design. Because the whole system trains together rather than in separate stages, conditioning on speaker identity or language became a matter of adding inputs to an already-unified model, not redesigning a multi-block pipeline to fit a new variable. That flexibility is why VITS turned into a reference point for the field. Later architectures tend to define themselves by how far they extend it, or by why they walked away from it.

The architectures that followed VITS: diffusion, flow matching, and codec language models

Three distinct families picked up where VITS left off, and they disagree, sharply, about what the next bottleneck actually is.

The diffusion camp treats speech generation as a denoising problem. Grad-TTS was an early application of diffusion probabilistic models to TTS, and NaturalSpeech 3 combined a factorized speech codec with factorized diffusion models to push toward high naturalness in neural speech synthesis. Diffusion buys quality, but it buys it with multi-step inference: generation happens across a series of denoising passes rather than in one shot. That's the toll booth on this road. Pay in latency, collect in fidelity.

Flow matching answered that latency cost head-on. Matcha-TTS and F5-TTS use conditional flow matching to reach diffusion-level fidelity with meaningfully faster inference, and OmniVoice is targeting omnilingual zero-shot synthesis using diffusion language models, chasing the same quality-versus-speed balance from a different angle.

The most conceptually different branch treats TTS as a language modeling problem instead of a signal generation problem. VALL-E and AudioLM discretize speech into tokens (VALL-E using neural audio codec tokens, AudioLM using a hybrid of semantic and neural audio codec tokens), then predict those tokens autoregressively with a transformer, the same basic recipe that powers text generation models. That framing produces convincing synthesis from remarkably short voice prompts. Systems built on it include CosyVoice, a two-stage autoregressive-then-diffusion pipeline trained on a large multilingual corpus; MaskGCT, a non-autoregressive masked generative model that drops explicit alignment entirely, also trained at large multilingual scale; and XTTS-v2, an open-source multilingual autoregressive model supporting 17 languages. The autoregressive members of this family inherit a familiar tax: generating tokens one at a time reintroduces the exact sequential latency that non-autoregressive models spent years trying to kill off. That's the whole tradeoff repeating itself under a new name. It's the whole tradeoff repeating itself under a new name.

Architectural Evolution and Its Impact on TTS Approach Choices for Developers

Raw audio quality has mostly stopped being the axis that separates these systems, and anyone still shopping on that axis alone is optimizing for a problem that's already solved. Independent listener studies from 2026 found that in blind tests, over roughly half of listeners could not correctly identify the synthetic speaker when leading TTS models were used. The competition has moved past "does it sound human" and onto expressiveness, controllability, and latency.

Latency in particular has stopped being a nice-to-have and started being table stakes. The TTS market is worth multiple billions of dollars and growing fast, with a large share of enterprise applications expected to build in AI agents, and for conversational deployments that shift makes sub-100-millisecond time-to-first-byte a baseline requirement rather than a premium feature. A voice agent that takes half a second to start talking doesn't read as slow. It reads as broken, the same way a phone call with a satellite delay feels broken even when every word eventually arrives intact.

That leaves a fairly practical map for choosing an architecture, and once the use case is fixed, the choice isn't close. Two-stage, non-autoregressive systems in the FastSpeech lineage are predictable and controllable, and they suit high-volume narration where prosody gets managed externally by a producer or a script: audiobooks and automated phone-response systems are the obvious fits. End-to-end generative models in the VITS and flow-matching lineage trade some of that predictability for better naturalness and variability, which suits expressive content creation, where one flat delivery would sound wrong on every listen. Codec language models are strongest at voice cloning and personalization from short prompts, but their autoregressive versions carry a latency tax that stops being academic the moment the use case is a live conversation instead of a file rendered once and left sitting on a server.

What VITS actually proved, and what has since propagated into most current production platforms, is narrower than it sounds: kill the seams between pipeline stages, and train every component against the same perceptual target instead of a proxy for it. Most modern systems run on that principle under the hood, regardless of what the marketing claims. Anyone evaluating a platform needs to ask which generative framework is actually doing the work behind that label, and its inference speed must match what the use case demands, not what the demo reel shows.