Addaly is in open beta. Things will change, and AI answers can be wrong — check anything that matters.

Making Things With AI

Images, video, voice and music — how they work, where they break, who owns them.

Lesson 50 of 848 min

From text to a waveform

The old pipeline, and why it sounded like that

For decades, text-to-speech had three stages. Convert text to phonemes with pronunciation rules and a dictionary. Predict a spectrogram — a picture of the sound's frequency content over time — from those phonemes. Convert the spectrogram to a waveform with a vocoder.

Each stage was separately engineered, and the joins were audible. Concatenative systems stitched together recorded fragments, which is why announcements at railway stations have that particular seam between words. Parametric systems produced the flat, buzzy voice everyone associates with a computer.

The quality ceiling came from the middle stage. Predicting a spectrogram from phonemes is an averaging problem: many real deliveries of a sentence are correct, and a model trained to minimise error over all of them produces the average of them, which is the flat one.

The current pipeline

The shift that changed everything was to treat audio as a sequence of discrete tokens, exactly as a language model treats text.

A neural audio codec — trained to compress audio and reconstruct it — produces a small vocabulary of tokens, typically at something like 50 to 75 tokens per second per codebook, with several codebooks stacked to capture successive levels of detail. A one-second clip becomes a few hundred integers.

Now the problem is familiar: given text, predict a sequence of audio tokens. A transformer does this, and it does it the way language models do — by sampling from a distribution rather than by averaging. That single change removes the flatness, because the model commits to one delivery rather than blending all of them.

The codec's decoder then turns the predicted tokens back into a waveform.

From text to a waveform, as it is done nowTextEverything aboutdelivery — thestress, thephrasing, wherethe breath goes— has to beinferred,because none ofit is in thewords.A languagemodel overaudio tokensA neural codecgives audio avocabulary:fifty toseventy-fivetokens a secondper codebook,severalcodebooksstacked, so asecond of soundis a few hundredintegers.It samplesrather thanaveragesThis is whatremoved theflatness. Themodel commits toone deliveryinstead ofblending everycorrect one, andthat is also whyit can skip aword or invent anoise.Codec decoderTokens back intoa waveform.A personlistens toevery lineGenerate line byline, re-rollthe failures,assemble inAudacity. Nobodygenerates tenminutes in onepass and shipsit.The old pipeline predicted a spectrogram from phonemes, which is an averaging problem: many deliveriesof a sentence are correct and the average of them is the flat one. Sampling is the whole difference,and it brings the whole failure family with it.
From text to a waveform, as it is done nowTextEverything about delivery — the stress, thephrasing, where the breath goes — has to beinferred, because none of it is in the words.A language model over audio tokensA neural codec gives audio a vocabulary: fiftyto seventy-five tokens a second per codebook,several codebooks stacked, so a second of soundis a few hundred integers.It samples rather than averagesThis is what removed the flatness. The modelcommits to one delivery instead of blendingevery correct one, and that is also why it canskip a word or invent a noise.Codec decoderTokens back into a waveform.A person listens to every lineGenerate line by line, re-roll the failures,assemble in Audacity. Nobody generates tenminutes in one pass and ships it.The old pipeline predicted a spectrogram fromphonemes, which is an averaging problem: manydeliveries of a sentence are correct and the averageof them is the flat one. Sampling is the wholedifference, and it brings the whole failure familywith it.
text -> (language model) -> audio tokens -> (codec decoder) -> waveform

What follows from the architecture

Three consequences that explain nearly everything users notice.

It hallucinates, because it samples. A generative sequence model can produce tokens that do not correspond to your text. In practice this appears as repeated syllables, a word skipped, an invented noise, or — the most disconcerting failure — the voice trailing off into unrelated babble. This is the same failure family as a language model repeating itself, and it is why any synthetic voice used in production must be listened to, every time, before it ships.

Temperature and sampling settings exist. Where a system exposes them, they behave as they do for text: low values give consistent, slightly duller delivery; high values give expressive readings with more failures. There is a genuine trade here, and for long-form narration the low setting plus a re-roll of the bad lines is usually the right one.

Long inputs degrade. Attention over a long token sequence is the same constraint as everywhere else in this course. Most systems handle a sentence or a paragraph well and drift over several minutes. Generating line by line and assembling in an editor is the standard practice, and it also gives you the ability to re-roll one bad line rather than a whole chapter.

The free tools, and what they can do

This is an area where open tools are genuinely competitive, which is not true everywhere in this course.

  • Piper — small, fast, runs on a Raspberry Pi, dozens of languages, good for narration where perfect naturalness is not required.
  • Several open codec-based systems produce voices close to commercial quality, run on consumer hardware, and support cloning from a short sample.
  • Audacity — free — for assembling, trimming and normalising the output, which you will need to do regardless of the generator.

The practical shape of a narration job is: generate line by line, listen to every line, re-roll the failures, assemble in Audacity, apply gentle compression, export. That is perhaps an hour for a ten-minute script, and most of the hour is listening.

The limitation worth remembering

The naturalness of current speech synthesis is real and it makes the failures more dangerous, not less. A flat robotic voice announced itself. A voice that is indistinguishable from a person, that occasionally inserts a word that was not in the script, and that is being used for something consequential — a medical instruction, a legal notice, an emergency announcement — is a different risk profile entirely.

The control is not technical. It is that synthetic speech in any consequential setting gets listened to by a person against the script before it is used. Every organisation deploying this at scale eventually learns that, usually after an incident.

The one thing to keep

Modern speech synthesis converts text into discrete audio tokens with a language model and turns those tokens back into sound with a neural codec, which is why it now sounds natural and why it hallucinates.

Before you move on

Why does current speech synthesis sound natural where the older spectrogram-prediction approach sounded flat?

Pick the one you would defend. Nobody sees your answer.

No ads. No data sale. No public scores on people. Ever.

© 2026 Addaly