Keying footage that was lit badly

The position you are usually in

You did not light the screen. Somebody filmed against a green sheet with one lamp, on a phone, and the sheet has creases.

Motion, Sound and Music makes the case that a key is decided on set. This lesson is the other half: what to do when it was not, and how to know when to stop.

What a keyer is doing

A chroma keyer builds a matte from colour: pixels close to the key colour become transparent, pixels far from it stay opaque, and pixels in between become semi-transparent.

Three consequences follow directly.

It needs colour resolution. At 4:2:0 the chroma channels are a quarter the resolution of the picture, so the matte is built from a quarter of the information. Edges come back stepped and unstable.

It needs separation. If the screen is unevenly lit — bright in the middle, dark at the edges, which is what one lamp produces — the "green" in the corner is a different colour from the green in the centre, and no single tolerance covers both.

It cannot distinguish spill from screen. Green light bouncing off the screen onto the subject's hair and shoulders is genuinely green, so the keyer takes some of the subject with it.

The repair workflow

In order, and each step should be small.

1. Garbage matte first. Draw a rough mask that excludes everything you do not need — the edges of the sheet, the light stands, the floor. Track it if the subject moves. This removes most of the unevenness before the keyer sees it, and it is the step people skip.

2. Even out the screen before keying. A qualifier or a soft window that lifts the dark corner of the screen toward the centre's brightness. You are not trying to make it look good, only to make it uniform enough for one tolerance.

3. Pull a core matte and an edge matte separately. Set the keyer aggressively to get a solid, clean interior — the core — and separately gently to preserve the semi-transparent edge. Combine them. Most keyers expose this as a clean plate, black point and white point on the matte.

4. Denoise before keying, not after. Noise in the chroma channels generates a crawling, unstable matte. Even a light temporal denoise on the source can transform the result.

5. Despill. Every keyer has a spill suppression control. It works by reducing green where green exceeds the other channels, and overdone it turns skin magenta and hair grey. Small amounts.

6. Match the composite. The subject and the background need the same black point, the same colour temperature, the same grain, and ideally a shared light direction. Most unconvincing composites fail here rather than at the matte — the edges are fine and the subject obviously does not belong.

Luma keys and difference mattes

When there is no green screen at all:

A luma key pulls a matte from brightness. It works for a subject against a bright sky or a dark studio, and only when the subject's own brightness does not overlap the background's.

A difference matte compares the shot against a clean plate — the same locked-off frame with nothing in it. Anything that differs is the subject. It works well for a locked-off camera and fails on any camera movement, changing light or noise. If you are also shooting, filming five seconds of the empty set at the end of every setup costs nothing and enables this.

When to stop

The honest thresholds:

• If the matte's edges are stepped and blocky on 4:2:0 footage and heavy blurring does not hide it, the shot will not key cleanly. Consider a soft-edged composite where the subject is deliberately blurred at the edge, a smaller subject in frame, or a different approach entirely.
• If the subject is wearing green, or has green in their hair or glasses, those parts will always go transparent. A separate hold-out mask is the only fix, and that is rotoscoping.
• If the screen is out of focus in a way that makes the edge ambiguous, the matte will be ambiguous.

In each of these, the correct answer is often to tell whoever asked that this shot cannot be keyed to the standard they want, and to propose the alternative — a different background approach, a reshoot of that one setup, or dropping the effect.

Free tools

Resolve's 3D Keyer on the colour page is in the free version and is genuinely good. Fusion's Delta Keyer is better still and is also free. Both expose the core/edge separation described above.

Kdenlive and Shotcut have chroma key filters that will handle a well-lit screen and will struggle with a badly lit one. OBS Studio has a live chroma key, useful when you are recording rather than editing.

Today

Find any green-screen clip, pull a key, and then compare the matte on its own against the matte after a garbage mask and a light denoise. The difference is the argument for doing the steps in order.