Why a second of video costs what it does

The naive arithmetic, and why it is wrong

A five-second clip at 24 frames per second is 120 frames. If video were images repeated, it would cost 120 times an image. That would be expensive and manageable.

It costs considerably more than that, and the reason is the part that makes video work at all.

A model that generated each frame independently would produce 120 unrelated pictures. To make frame 47 continue frame 46, the model must see other frames while generating each one. That means attention across time as well as across space, and attention is quadratic: doubling the number of frames quadruples the cost of the temporal attention layers.

Current video models handle a fixed window of frames jointly. Within that window, every latent position can attend to every other, across both space and time. A 121-frame clip at a 60 by 104 latent resolution has on the order of 750,000 positions attending to each other. That is the number that decides what hardware you need.

Where the memory goes

The practical consequence is that video generation is memory-bound rather than time-bound. You can wait longer for a slow generation. You cannot wait your way past running out of video memory.

Rough current figures for open video models: a small one generating a few seconds at low resolution fits in 8 to 12 GB with quantisation; the larger open models want 24 GB and gratefully accept more. This is why video generation went to hosted services for most people first, while image generation went local.

The mitigations are the same family as for images, applied harder:

• Quantisation of the weights.
• Tiled decoding, because the video autoencoder's decode step is the single largest memory spike.
• Fewer frames per pass, generating in chunks and joining them.
• Lower resolution, then upscaling each frame afterwards — which works, and introduces its own flicker if the upscaler is not temporally aware.

The video autoencoder

Images use a spatial autoencoder that compresses by eight in each direction. Video autoencoders compress in time as well, typically by four: four input frames become one latent frame. A 121-frame clip becomes about 31 latent frames.

This is why video models produce frame counts in odd-looking numbers — 49, 81, 121 — rather than round ones. The counts are set by the temporal compression factor plus one.

It also introduces a specific artefact. Because four frames share one latent, fast motion within those four frames has to be reconstructed by the decoder from a single compressed representation. Rapid movement therefore comes back smeared or stuttering in a way that is not present in the latent itself. If your clip has a fast pan or a quick gesture and the result looks like a badly encoded video file, this is the mechanism, and generating at a higher frame rate does not fix it because the compression factor is fixed.

What this means for how you work

Three practical consequences follow directly from the arithmetic.

Generate short. The cost curve punishes length superlinearly, and the drift problem covered later in this module punishes it again. Five seconds is not an arbitrary limit; it is where several curves cross.

Generate small, upscale after. Resolution multiplies into the same attention cost. A 480-pixel generation upscaled to 1080 costs a fraction of a native 1080 generation and is frequently indistinguishable after grading.

Expect to throw work away. Because each generation is expensive, the temptation is to accept a nearly-right clip. The professional habit is the opposite: decide in advance what "usable" means for the shot, and discard anything that is not, because a flawed clip costs you far more in the edit than a regeneration costs in credits.

The honest limitation to carry: none of this is a temporary inefficiency waiting for better engineering. The quadratic term is in the architecture. Approaches that reduce it — attending only to nearby frames, or compressing time harder — reduce the cost and give up exactly the long-range consistency that the attention was buying. That trade is the subject of the next lesson.

It is worth grounding the cost in something concrete, because "expensive" is not a number. On hosted services, generated video is currently priced in the region of a few cents to a few tens of cents per second of output, depending on model and resolution. Against a hit rate of one usable shot in six or eight attempts — which is a realistic figure for text-conditioned work and improves considerably with a first frame — a usable five-second shot lands somewhere between the price of a coffee and rather more. Multiply by the thirty shots a two-minute piece needs and you have the real budget. Doing that arithmetic before starting is the difference between a project that finishes and one that stops halfway.