SSML and Markup Control in Neural TTS Systems

The most useless sentence in TTS documentation is "results may vary." Results always vary; that's the whole problem plain text creates in the first place. Feed the same sentence to the same model twice and the pauses land in different places, because plain text gives the engine nothing to work with beyond the words themselves. SSML addresses that by making pacing, emphasis, and pronunciation into instructions the engine has to follow rather than guesses it gets to make. This matters most where a wrong number turns into a support ticket instead of just an aesthetic gripe: IVR menus, voice assistants, accessibility tools, anywhere a misread digit costs someone money or time. What follows covers what SSML actually controls, where vendors quietly bend the spec to their own purposes, and where markup runs out of road and the neural model takes the wheel.

SSML descends from JSML and SABLE, two markup efforts from 2000, and the W3C folded their ideas into SSML 1.0 in 2004, then updated it to SSML 1.1 in 2010. It's XML-based, so it looks like the markup people already associate with old enterprise software: angle brackets, nested tags, a strict document structure. That lineage is part of why it stuck around, and it's boring in the way infrastructure is supposed to be boring, which is a compliment.

The core tag system and what each tag actually controls

Every SSML document opens with a <speak> tag, and everything else nests inside it as the container the other tags do their work in.

<break> inserts a pause, either by exact duration or by a strength value like "medium" or "strong." It's the simplest tag in the spec, and also the most reliably supported across engines, probably because a pause is a pause no matter who's rendering the audio.

<prosody> adjusts pitch, rate, and volume, and it's the widest set of levers a developer gets over how something sounds. It's also the tag most likely to behave differently between vendors, since "significantly faster" doesn't map to the same acoustic result across different underlying models.

<say-as> tells the engine how to read structured data. A string of digits can come out as a phone number, a date, a currency amount, or a plain cardinal number, and without this tag the engine guesses based on context clues that don't always exist. An IVR system reading back an alphanumeric order code needs <say-as interpret-as="characters"> so it spells the code out instead of trying to pronounce it as a word. A billing prompt handling multiple currencies needs <say-as interpret-as="currency"> so "45.00" doesn't come out as forty-five of whatever currency the model defaults to.

<phoneme> drops to the phonetic level, using IPA or CMU ARPAbet symbols to spell out exactly how a word should sound. This is the escape hatch for when the engine's grapheme-to-phoneme model keeps getting something wrong: a proper noun, a brand name, a word borrowed from another language. <sub> solves a lighter version of the same problem: it swaps a spoken alias in for written text. An education app that needs "H₂O" to come out as "water" needs substitution, not phonetic spelling.

<emphasis> marks a word or phrase for stress, though as the next section covers, neural engines don't all honor it the same way. <voice> switches the speaker mid-document, which makes multi-character dialogue possible inside one SSML pass instead of stitching together separate audio files.

One more piece worth knowing: the PLS standard lets a developer define pronunciations once, at the document or account level, instead of repeating <phoneme> tags in every script. Amazon Polly and Microsoft Azure both support custom lexicons this way. If a company name gets mispronounced across a hundred prompts, fixing the lexicon fixes it everywhere at once, instead of fixing it a hundred times.

How vendor SSML implementations diverge from the W3C spec

The W3C spec defines what tags exist. It doesn't force any vendor to honor all of them, and that gap is where most production bugs actually live.

Divergence shows up three ways: tags a vendor skips entirely, proprietary tags a vendor bolts on top of the spec, and tags that exist everywhere but behave differently depending on whose engine renders them. As of mid-2026, Google restricts <emphasis> and <prosody pitch> on its Studio and Chirp voices, and Amazon Polly restricts <emphasis> on its neural voices. Both tend to strip the tag or ignore it rather than throw an error, and that's the worse outcome: a silent failure is harder to catch in testing than a loud one, and it usually surfaces in production instead of in a code review.

Then there's the other direction, extensions that add capability the base spec never promised. AWS ships <amazon:effect> for things like whispered speech. Microsoft Azure ships <mstts:express-as> for speaking style and role-play delivery. Both buy expressiveness at the cost of portability; markup written for one engine's extensions won't render the same way, or at all, on another.

Azure's current lineup includes its Neural HD V2 voices, built on the DragonHDLatestNeural base model, alongside a Dragon HD Omni model that does context-aware emotion detection, paired with more than 140 languages and over 500 voice options. That puts Azure's extension surface ahead of most competitors on raw breadth. Google Cloud TTS covers similar ground with its WaveNet and Neural2 models and solid support for pitch, rate, and pronunciation, but its extension surface is narrower than Azure's. Amazon Polly offers more than 100 voices across 40-plus language variants, with strong <say-as> and lexicon support, alongside the neural-voice prosody restrictions already mentioned. IBM Watson TTS rounds out the field with SSML support for pitch, speed, and pronunciation over both REST and WebSocket, including real-time streaming, a reasonable pick for teams already running IBM infrastructure who don't want a second vendor relationship just for voice.

Here's what most teams get backwards: they build against the full spec and assume it travels. The "safe" SSML subset that behaves the same everywhere is a lot smaller than the full spec, and teams building for multiple engines should map the intersection of what's supported, rather than the union of what's possible. Reach for the union instead, and something breaks in production, usually the one tag nobody bothered to test because it worked fine on the first engine.

Where SSML ends and neural model inference takes over

SSML operates at the level of tokens and segments. It can set a pause to exactly 400 milliseconds, and it can spell out a phoneme string for a word the model keeps butchering. What it cannot do is tell a model to sound empathetic, or tense, or casually amused, because there's no tag in the spec for a feeling, and there never will be.

Everything SSML leaves unspecified gets filled in by the model's learned priors: the shape of the pitch contour across a sentence, the micro-timing between words, where a breath lands, the emotional color of a phrase. That's inference, and it marks a structural ceiling rather than a missing feature waiting on the next spec revision. No tag turns a flat voice warm, or makes a fast voice sound urgent instead of just rushed; that work happens inside the model or it doesn't happen at all.

A handful of mechanisms operate below or beside SSML to handle exactly this problem. Style tokens and prosody transfer copy a general tone from a reference audio clip, while variance adaptors, the approach used in FastSpeech 2, give direct control over pitch, duration, and energy at the level of individual audio frames. Diffusion-based models like StyleTTS 2 let style emerge from a learned latent space instead of an explicit control knob. Codec language models like VALL-E clone a speaker's voice and emotional register from a few seconds of reference audio, with no fine-tuning step required.

Successive codec language models continue pushing in this direction, targeting faster inference and more stable decoding. That's where model-side control is heading: less explicit markup, more learned behavior pulled from a short reference clip.

The split that matters in practice: SSML handles correctness, numbers read right, codes spelled out, dates formatted for the locale. Neural style control handles expressivity: tone, warmth, character voice. They aren't competing for the same job, which is exactly why testing has to cover both separately. An SSML regression, where a tag silently stops rendering, and a perceptual regression, where the model's default tone drifts after an update, are two different failure modes that show up in two completely different places in a QA process, and catching one tells you nothing about the other.

Building a production TTS pipeline that combines both layers

A production pipeline needs three layers stacked on top of each other: SSML plus custom lexicons for correctness and reproducibility, neural style selection or reference audio for expressivity, and validation plus version control to hold the whole thing together so it doesn't drift silently over time.

The first real architectural fork is streaming versus batch. Streaming TTS returns audio incrementally, which matters for voice agents and anything interactive, where a user sitting through a two-second delay before the first word plays feels that wait in their bones. Batch synthesis renders the whole file before returning it, fine for audiobooks, podcasts, or pre-recorded IVR prompts where nobody's waiting on the other end of a call.

For real-time agents, latency has a real number attached to it, not a vibe. Early speech-to-speech systems like Moshi hit 160 milliseconds of end-to-end latency using single-step processing, and that number has become close to a floor for what a conversational voice pipeline needs to hit before it stops feeling natural and starts feeling laggy.

Governance is the less glamorous half of this, and it's the half teams skip until something breaks. SSML documents should live in version control right next to the voice configuration they're paired with, because an engine update can change how a tag gets interpreted even when the markup itself never changed. Custom lexicons deserve the same treatment: one shared PLS file for domain terms, rather than <phoneme> tags scattered across fifty scripts. When a pronunciation needs fixing, fixing it in one place beats hunting through every document that mentions the word.

There's a deployment pattern showing up specifically in healthcare and finance: keeping audio generation inside the organization's own infrastructure instead of sending it out to a managed API. Regulated industries increasingly lean this way, and once that decision gets made, SSML portability across whatever self-hosted engine gets chosen stops being a nice-to-have and becomes a hard requirement.

For teams running a full conversational stack, speech-to-text into a language model into text-to-speech, the SSML layer sits at that last step. The language model's output can be templated to emit SSML directly, turning the LLM into a controllable bridge that hands off structured instructions to the voice layer instead of raw, unformatted text.

How to evaluate a TTS API against these control requirements

The market context matters here. TTS was valued at $4.8 billion in 2025 and is projected to hit $35.3 billion by 2035, growing at a 22.4% compound annual rate, and more vendors means more feature claims and more marketing copy that all reads the same after the third landing page. Structured evaluation gets more necessary as the field gets more crowded, not less.

Five things are worth checking, in this order. First, SSML compliance: test the portable subset, <break>, <say-as>, <phoneme>, <prosody rate> and <prosody volume>, before touching any vendor-specific extension. Note what gets silently dropped versus what throws an actual error, because those are two very different debugging experiences six months into a production deployment.

Second, neural expressivity. Test with real content pulled from the domain the application actually serves, rather than the cheerful demo sentences vendors put on their landing pages, and check whether the model's default prosody matches what the use case needs: conversational, authoritative, empathetic, whatever the register calls for.

Third, latency. Check streaming availability and time-to-first-audio under realistic payload sizes, not toy examples. For agent-style use cases, sub-200-millisecond latency is close to a practical floor below which the interaction starts to feel sluggish.

Fourth, language and voice breadth, especially for anything multilingual. Native-quality output across the target languages matters; a transliterated approximation that technically produces the right words in the wrong accent isn't the same product, no matter what the spec sheet says.

Fifth, deployment model. Cloud API versus on-premise or self-hosted matters a lot for regulated industries, and SSML portability toward a self-hostable engine, open-weight options like Qwen3-TTS under an Apache 2.0 license are one example, becomes the deciding factor once data sovereignty outweighs the convenience of a managed service.

Pricing deserves a sanity check too, if only because the spread is wide enough to be disorienting. ElevenLabs, a text-to-speech and voice cloning platform, sits in the middle of this range with tiered API pricing. Speechmatics prices around $0.011 per 1,000 characters, while some managed API providers run as high as $30 per million characters. Comparing cost per character in isolation, without weighing it against quality and feature differences, tells you almost nothing; it's like comparing two cars by the price of gas alone.

Pick the engine whose SSML compliance, model quality, and deployment model line up with what the application needs to get right and how expressive it needs to sound, since both layers have to pass the test and neither one gets to coast on the other's strength.