How a machine writes a waveform

Audio is a lot of numbers

A stereo track at CD quality is 44,100 samples per second per channel. Three minutes is about 16 million numbers. Predicting those directly, one at a time, was tried and is impractical for anything beyond a few seconds.

The solution is the same one the voice module described, applied at higher quality. A neural audio codec learns to compress audio into a small stream of discrete tokens and to reconstruct it. A common configuration produces tokens at 50 per second, with four to eight stacked codebooks, so one second of music is a few hundred integers rather than 88,000 samples.

The stacking matters and is worth understanding. The first codebook carries the coarse structure — roughly, the shape of the sound. Each subsequent codebook encodes the residual error left by the previous ones, adding detail. This is residual vector quantisation, and it means you can trade quality for speed by generating fewer codebooks, which is exactly what fast preview modes do.

Given that representation, generation is sequence prediction: a transformer predicts the next tokens conditioned on a text description, a melody, or the tokens so far. The codec decoder turns the result back into sound.

The other approach, and why both survive

Some systems work on spectrograms instead — the time-frequency picture of a sound — treating them as images and applying diffusion. A vocoder converts the spectrogram back to a waveform.

This inherits the strengths of image diffusion: strong texture, parallel generation, good conditioning tools. It also inherits the vocoder problem, because a spectrogram discards phase information, and reconstructing phase is where the characteristic metallic or watery artefacts in some generated music come from.

Current strong systems are frequently hybrids. Which one you are using is usually stated in the model card, and it predicts the artefacts: token-based systems tend toward abrupt glitches and structural drift, spectrogram-based ones toward smearing and phasiness.

The numbers that govern quality

Codec bitrate is the ceiling. No amount of model capacity produces audio better than the codec can reconstruct. The test from the image module works identically here: run a real track through the codec's encoder and straight back out, with no generation at all. What comes back is the best the system can ever produce. Do this once and you will stop expecting master-quality output from a model whose codec cannot deliver it.

Sample rate. Many open music models generate at 32 kHz rather than 44.1, which means nothing above 16 kHz. On a phone this is inaudible. On good headphones the top end sounds slightly closed, and it matters if the track will be mastered alongside recorded material.

Token rate times duration is the cost, which is why generated tracks are typically 30 seconds to a few minutes and why extending is done in chunks with the same drift problem as video.

What free tools exist

The open end of this field is genuinely usable:

• Open music-generation models run on consumer hardware and produce instrumental tracks of reasonable quality. Licences vary considerably — several are non-commercial only, which is a detail people miss until it matters. Read the licence before using output in paid work.
• Some open models are trained specifically on permissively licensed and public-domain audio, which materially changes the rights position and is worth seeking out.
• Audacity for editing, LMMS and Ardour for full production, all free.

Where the quality actually is

Being specific rather than dismissive: current generated music is good at texture, atmosphere and short loops. A four-bar bed, an ambient wash, a percussion loop, a sting — these are frequently indistinguishable from library music and take seconds.

It is weak at everything that depends on structure over time, which is the next lesson, and it is weak at anything with a lead vocal that has to be intelligible, expressive and in tune across a whole song.

There is one more setting worth finding in whatever tool you use: the number of codebooks or the quality tier. Fast preview modes generate fewer residual codebooks, which is why the draft sounds thin and grainy and the final render does not. People frequently judge a model on its preview output and conclude it is poor. Generate one full-quality render before forming an opinion, and use the preview for what it is good for, which is deciding whether the idea is right.

The limitation to carry: audio quality has a hard ceiling set by a component most users never look at, and structure has a soft ceiling set by the same attention-window problem as every other medium in this course. Neither is a matter of prompting better.