Tracking a point, a plane and a camera

Three different problems

Point tracking follows one small feature through a shot and gives you its position over time. Use it to attach a label to a moving object, or to drive a mask.

Planar tracking follows a flat surface and gives you its position, rotation, scale and perspective. Use it to replace what is on a screen, a sign or a wall.

Camera tracking — also called match moving — solves for the three-dimensional path the camera took through the scene, so you can place an object into the space and have it stay put. Use it when something has to sit in the world rather than on a surface.

They are progressively more expensive and progressively more capable. Choose the cheapest one that solves your problem.

Point tracking, and what makes a good feature

A tracker searches a small region of the next frame for the pattern it found in the current one. It works well when the pattern is distinctive and fails when it is not.

Good features: a corner, a high-contrast junction, a small dark mark on a light surface, the corner of a window.

Bad features: a point along a straight edge (it can slide along the edge freely — this is the aperture problem, and it is why edges track badly while corners track well), a repeating pattern, a soft gradient, anything out of focus, anything with motion blur.

When a track fails mid-shot:

• Track backwards from a later frame.
• Split the shot and track each part.
• Move the tracker to a different feature that is rigidly attached to the same object, and offset the result.
• Keyframe by hand through the difficult section.

Planar tracking

The technique for screen replacement, sign replacement, and any mask attached to a flat surface. It tracks the whole surface rather than individual points, so it survives partial occlusion — a hand passing over a phone screen does not break the track.

Mocha is the paid standard. Fusion's Planar Tracker, inside free Resolve, does the same job. Blender's motion tracker also handles planar cases.

The workflow for replacing what is on a screen:

1. Track the plane of the screen across the shot.
2. Attach a corner-pin transform to the track.
3. Put your replacement image through the corner pin.
4. Mask out anything that passes in front — a hand, a head. This part is rotoscoping and it is where the time goes.
5. Match the composite: add the screen's reflections, its brightness relative to the room, a little blur if the shot is not sharp, and grain.

Step 5 is what separates a convincing replacement from an obvious paste. A real screen has glare and is rarely perfectly sharp.

Camera tracking

Solving the 3D camera path requires parallax — the camera must actually move through space, not merely pan. A pure pan from a tripod has no parallax and cannot be solved for position, only rotation.

It also needs the scene to be rigid. Trees in wind, water and crowds all contribute points that move independently, and they degrade the solve.

Blender's motion tracker is free and genuinely capable. The workflow: detect features, solve the camera, check the solve error — under about 1 pixel is good, over 3 is unusable — then place 3D objects in the solved scene and render with a transparent background to composite back.

Fusion has a Camera Tracker as well, in free Resolve.

Budget for this: a straightforward solve on a well-shot moving shot is an hour. A difficult one can consume a day and still fail.

Helping the track at the shoot

If you also film, three things make tracking dramatically easier and cost nothing:

• Keep the shutter fast enough to limit motion blur on tracking shots. Blurred features do not track.
• Put tracking markers in the scene — small pieces of tape at known positions on a blank wall or a green screen. They give the tracker features where there are none, and they are removed later with the techniques in the next lesson.
• Move the camera. For a 3D solve, a slow lateral move gives far more parallax than a pan from the same spot.

Knowing which one you need

• Label follows an object in 2D → point track.
• Replace what is on a flat surface → planar track.
• Put an object into the room so it stays where you put it → camera track.
• Mask follows a rigid object → point or planar track.
• Mask follows a person who changes shape → not a track; that is rotoscoping.

Today

Track a single point in a moving shot and attach a text label to it. Watch where the track slips. That is what a tracker finds hard, learnt in five minutes.