What a voice print actually is

A few hundred numbers

A speaker encoder is a network trained on a task that sounds unrelated to synthesis: given two audio clips, say whether they are the same person. Trained on many thousands of speakers, it learns to produce a vector — typically 192 to 512 numbers — that is close for two recordings of one person and far apart for two people.

That vector is the voice print. It captures the things that make a voice recognisable: fundamental pitch range, the resonances of the vocal tract, the characteristic timing and attack.

To clone a voice, you compute that vector from a sample and use it as conditioning while generating. No training run, no fine-tuning, nothing stored beyond the vector. This is called zero-shot cloning and it is why the whole thing takes seconds.

How little audio is enough

The numbers matter here because people's intuitions are badly calibrated.

• Three seconds is enough for a recognisable clone with early research systems, and current ones do better.
• Ten to thirty seconds of clean speech produces a clone most listeners cannot distinguish from the source in a short clip.
• Several minutes, used for actual fine-tuning rather than zero-shot conditioning, produces a clone that holds up across long-form narration and unusual phrasing.

Quality of the sample matters far more than quantity. Clean, close, single-speaker, no music, no reverb, one consistent emotional register. A five-minute sample from a noisy video call produces a worse clone than fifteen clean seconds.

The uncomfortable consequence: everyone who has ever posted a video with their voice in it, left a voicemail greeting, appeared on a podcast, or spoken in a recorded meeting has published enough material. There is no meaningful way to withhold it, because the amount needed is below the threshold of ordinary social participation.

Fine-tuning versus zero-shot

Worth distinguishing, because the consent and security implications differ.

Zero-shot conditions on a vector at generation time. Nothing persists. The sample can be discarded. This is what a service does when you upload a clip and get speech back immediately.

Fine-tuning trains model weights on a person's recordings, producing a durable artefact — a file that is, in a practical sense, a copy of a voice. It sounds better and it is a thing that exists, can be copied, shared, sold and stolen. Any agreement about a fine-tuned voice needs to say who holds the artefact, where it is stored, who may use it, and what happens to it when consent is withdrawn. "We deleted the recordings" is not the same as "we deleted the model", and a contract that only mentions the recordings has missed the point.

Why watermarking the output is not a control

Several services embed an inaudible watermark in generated speech. This is worth doing and it does not solve the problem, for reasons the provenance module covers in general: the watermark identifies audio that came from that service, and open models with no watermark exist and run on a laptop. A control that only binds the compliant is a useful record and not a defence.

What to do about your own voice

Three things, none of them heroic.

Assume it is clonable and act accordingly. Do not treat voice as an authentication factor for anything of yours, and ask your bank whether they do. Some do, and you can usually opt out.

Agree a verification method with the people who matter. A word, a fact, a callback number. This is the control that actually works, and it is covered in full at the end of this module.

Get the terms right when you consent. If you record for a client, read what the contract says about synthetic reproduction. A release written before 2022 almost certainly does not address it, and silence is not protection — it is an argument waiting to happen.

The honest limitation of all advice here: none of it prevents anyone from cloning your voice. The technology is distributed, the samples are public, and the barrier is not technical. What the advice does is remove the value of doing it, by ensuring that nobody who matters to you will act on a voice alone. That is a smaller claim than people want and it is the true one.