When a reference image beats any sentence

The bandwidth argument

A 75-token prompt carries, at most, 75 vectors of a few hundred numbers each, and much of that capacity is spent on saying which objects exist. An image encoder converts a reference picture into a comparable set of vectors — but those vectors describe an actual arrangement of colour, light, texture and geometry rather than a verbal gesture at one.

This is why the single largest improvement most people can make to their results is not a better prompt. It is providing a picture.

There are several distinct ways to do it, and confusing them causes most of the frustration.

Four different things called "reference"

Style reference. An adapter encodes the reference image and injects it alongside the text conditioning, so the output takes on the reference's palette, texture and rendering without copying its content. This is what IP-Adapter and the various "style reference" or "sref" features do. Good for: matching an existing set of illustrations, keeping a brand look.

Structure reference. The reference is converted into an intermediate map — edges, depth, human pose, a rough segmentation — and the model is conditioned on that map at every step. This is ControlNet and its relatives. Good for: keeping a composition, redrawing a rough sketch, putting a new subject into an existing layout. Content and style come from your prompt; only the geometry comes from the reference.

Image-to-image. The reference is encoded into a latent, noised part-way, and denoised from there. The output stays close to the original in proportion to how little noise was added. Good for: variations, a repaint, a change of medium.

Subject reference. The reference supplies a specific identity — a face, a product, a character — which the model then places in new scenes. This is the hardest of the four and still the least reliable.

Choosing the wrong one produces the classic complaint that "the reference did nothing". A style adapter will not preserve the composition. A depth map will not preserve the colour. Neither will preserve a face.

The practical routine

For most jobs the fastest path is not a better description but this:

1. Find or make an image with the composition you want. A phone photograph, a scribble in GIMP, a stock image, anything.
2. Extract a depth map or edge map from it. Free tools do this in one click and it takes under a second.
3. Prompt for the content you want, conditioned on that map.

The composition is now yours, exactly, and the prompt only has to carry subject and style — which is comfortably within 75 tokens.

Anyone who has spent an afternoon writing "the woman is on the left, slightly behind the table, looking away from the camera" and getting it wrong every time will find this a relief. Spatial language is exactly what these encoders are worst at, and it is exactly what an image conveys for free.

What you owe the source

A reference image is somebody's work unless you made it. Using a photograph as a depth map extracts geometry, which is a weaker taking than reproducing the picture — but "weaker" is not "none", and the law here differs by country and is unsettled, which the final module covers properly. Two habits are worth adopting regardless of what any court eventually decides.

Keep a record of where every reference came from. If a client later asks, "is anything in this derived from someone else's image", you want to be able to answer precisely rather than from memory.

And prefer references you own or that are openly licensed. Your own photographs, images from Unsplash, Pexels or Wikimedia Commons under their stated terms, or a rough sketch you drew yourself in Krita. A sketch you drew badly works as well as a photograph for structure conditioning, because the model only reads the edges.

The honest limitation: structural conditioning constrains geometry, not meaning. A depth map of a person will get you a person-shaped thing in that pose. It will not stop the model producing four fingers, and it will not carry an expression. Reference images make composition solvable. They do not make everything solvable.

There is one more caution, about strength. Every reference method has a weight, and every one of them fails in the same two ways at the ends of its range. Too strong, and the output is the reference with a light coat of paint — recognisably the source, which is the case most likely to cause a rights problem as well as a boring image. Too weak, and you have paid for a conditioning step that did nothing. The useful setting is nearly always somewhere in the middle third, and the only way to find it for a given model is a short sweep with everything else fixed. Do that sweep once per method and write the number down.