Guidance: the dial that makes it obey

Two predictions, one exaggerated difference

Left alone, a diffusion model trained on captioned images follows a prompt loosely. The prompt is one influence among many, and early systems produced images that were plausible but only vaguely on topic.

The fix, classifier-free guidance, is a trick of arithmetic rather than training. At each step the model is run twice: once with your prompt, and once with an empty prompt. That gives two noise predictions — where the model would go with your words, and where it would go without them. The direction from the second to the first is "what your prompt is doing". Guidance amplifies that direction and follows it past where either prediction actually pointed.

guided = unconditional + scale * (conditional - unconditional)

At scale 1 the guided prediction equals the conditional one and the prompt is a gentle influence. At scale 7 you travel seven times the difference. At scale 20 you are far outside the range the model was trained to produce anything sensible in.

Two facts follow immediately, and they explain most of what people notice. First, guidance is why generation costs roughly twice the compute of a single forward pass — the model runs twice per step. Second, the "negative prompt" box in most interfaces is not a special feature. It replaces the empty prompt in that second run. You are not telling the model to avoid something; you are moving the point it is being pushed away from.

What each end of the dial looks like

Typical defaults sit at 6 to 8 for the Stable Diffusion family. Walking the range on a fixed seed teaches it in one minute:

• 1 to 3. Loose, painterly, often ignores half the prompt. Colours are muted and natural. Occasionally the most photographic-looking result, because real photographs are not saturated.
• 6 to 9. The usable band for most checkpoints. Prompt followed, contrast normal.
• 12 to 16. Prompt followed hard. Colours push toward the edges of the range, skin goes waxy, highlights clip to flat white.
• 20 and above. Burnt. Posterised colour, hard black outlines, sometimes a rainbow shimmer in flat areas.

The burning is a direct consequence of the arithmetic. Extrapolation drives the predicted latent outside the distribution the VAE was trained to decode, so the decoder is asked to render values it has never seen. Clipped, over-saturated output is what an autoencoder does when handed impossible inputs.

The models where the dial has moved

Distilled and flow-matching models have changed the numbers, and advice written for one family is actively wrong for another. Several current models bake guidance in during training and expose a differently scaled parameter, where the useful band is 1 to 5 and setting it to 7 produces the burnt look that 15 produces elsewhere. Turbo and Lightning variants are trained to run at guidance 1 with three or four steps, and raising it destroys them.

There is no way to read the right number off the interface. It is on the model card, in one line, and reading that line is the difference between a model that seems broken and a model that works. This is a good habit to build early: before blaming a checkpoint, check the three numbers its author published — native resolution, step count, guidance scale.

There is also a scheduling trick worth knowing, because several free tools expose it. Guidance does not have to be constant across the run. Applying it strongly in the early steps, where composition is decided, and weakly in the late steps, where texture is laid down, gives prompt obedience without the burnt surfaces. Tools call this guidance rescaling, dynamic thresholding or CFG scheduling depending on who wrote them; ComfyUI ships several nodes for it and the WebUI has it behind an extension. If you find yourself stuck choosing between a picture that ignores you and a picture that looks like melted plastic, this is the setting that resolves the trade-off, and it is free.

When lowering it is the answer

Most people only ever raise guidance, because raising it makes the model obey. It is worth knowing the failure that lowering it fixes.

If your images look like stock illustration — plastic skin, impossible contrast, every surface glossy — the cause is usually guidance, not the prompt or the model. Photographs contain dull greys and blown-out corners. High guidance drives every region toward the most prototypical version of what it is, and the most prototypical mug is glossier than any real mug. Dropping from 9 to 4 and adding two steps often produces the image people were trying to reach with another twenty words of "photorealistic, 8k, award-winning". Those words are doing far less than the dial.