Video With AI
Plan the shots, generate ten takes, throw eight away, and cut what is left in a real editor.
- Level
- Some background helps
- Lessons
- 66
- Reading time
- 587 min
- Price
- Free, no sign-up to read
Generative video taught as production work rather than a demo reel. Why clips are a few seconds long and what that does to your shot list, why image-to-video is the professional default, how to direct a camera move instead of describing a mood, what consistency across shots actually costs, the consent and labelling line on faces and voices, and how to price an afternoon of generating before you start.
Opens after the Editing Video exam
Sign in, finish that course, and pass its exam. You can read this syllabus meanwhile.
Go to Editing VideoModule 1
What the machine is actually doing
Before the shot list, the machinery. A video model compresses your clip roughly fifty times, denoises the whole block of time at once, and attends across every patch of every frame simultaneously. This block walks that path end to end — encoder, schedule, attention, decoder — and turns each stage into something you can act on: which setting changes the picture, which only changes the bill, why the last second loosens, and what it takes to run any of this on hardware you own.
By the end you can
Trace a prompt and a first frame through a latent video model — encoder, noise schedule, attention across time, decoder — and use that path to predict which settings change the picture, which change only the cost, why the final second degrades, and what VRAM a given model needs before you download it
- 1Nobody is hiding the long version from youClip length is a memory-and-attention limit, not a product tier, so plan in shots and cut before the drift rather than trying to generate through it.
- 2Your clip is compressed fifty times before anything happensA video autoencoder throws away roughly ninety-eight per cent of the pixel data before generation begins, so any detail finer than an eight-pixel block is being reinvented by the decoder rather than preserved.
- 3The whole clip is made at once, not frame by frameA video model denoises every frame of the clip simultaneously across a fixed number of steps, which is why nothing can be streamed out early, why step count hits diminishing returns fast, and why distilled turbo models trade motion range for speed.
- 4Why a chair stays a chair and a face does notAttention lets every patch of every frame see every other, which is what holds a scene together, but it costs the square of the token count and carries no object identity — so coherence is an emergent statistical effect, not a guarantee about any particular thing.
- 5The prompt vocabulary was written by whoever captioned the training clipsVideo models learned from automatically captioned clips filtered for aesthetics and motion, so the words that work are the words those captions used, and the model's defaults are the defaults of stock footage.
- 6Same settings, different clip, and why that is not a bugA seed fixes the starting noise, but reproducibility also requires the same weights, precision, batch shape and kernels — which is why local runs repeat exactly and hosted APIs usually will not promise it.
- 7The four numbers, and which of them you should touchGuidance scale controls how hard the model is pushed toward your prompt and away from its unconditional prediction, so raising it past a narrow band burns contrast and freezes motion rather than improving adherence.
- 8Generated at sixteen, delivered at twenty-fourMany models generate at a low native frame rate and are interpolated up afterwards, which is cheap and usually invisible — except at occlusions and fast motion, where interpolation produces a specific smeared artefact worth recognising.
- 9The ranking will be wrong by the time you read thisBenchmark models on one hard shot from your own list, because public rankings measure which clip looks nicer, not whether it did what you asked.
- 10What it takes to run one on your own cardA video model's VRAM requirement is weights plus activations plus the decode, and quantisation and offloading trade speed for fit — which is why a 14-billion-parameter model can run on 8GB slowly or not at all on a phone.
Module 2
The first frame, and everything decided in it
In image-to-video the still is the contract: framing, light, wardrobe, style and every flaw persist through the clip and then degrade. This block treats the frame as production work rather than a prompt result — composing for a move that has not happened yet, fixing the subject by hand in a free editor, compositing a real object in before any motion exists, controlling where the shot lands with an end frame, getting a camera move with no video model at all, and reading a frame in advance for the second it will fail.
By the end you can
Produce a first frame at the model's native ratio and resolution with a stated light direction, a hand-repaired subject and any real object already composited in, predict from that frame alone which part of the clip will fail, and store it with enough provenance to rebuild the shot six weeks later
- 11The still frame is where the decisions belongEvery decision you can make in a still costs a hundredth of what it costs to make in a clip, so fix the frame before you ever spend on motion.
- 12Compose for the frame you will end onA camera move changes the framing, so the still has to be composed for where the shot ends rather than where it starts, and every edge of the frame is a place where the model has to invent what enters.
- 13One light direction per scene, written downLight direction is the strongest continuity signal an audience reads and the easiest one to lose between generations, so it should be decided once for a scene, written down, and put into every first frame before any motion is generated.
- 14Fix the frame yourself instead of re-rolling itRepairing a still by hand is deterministic and costs one attempt, while re-generating is a fresh random draw that may fix one flaw and introduce two, which is why professionals reach for an editor long before they reach for the regenerate button.
- 15Put the real object in before you animate itA model has no reference for your client's specific product or person, so the professional split is to generate the environment and the camera move and composite the real thing into the still — because a nearly-right label reads as a counterfeit rather than a stylisation.
- 16Deciding where the shot landsSupplying both a first and last frame turns generation from an open-ended sample into an interpolation between two fixed points, which is the most underused control in the category and the one that fails hardest when the two frames are too different.
- 17A camera move with no video model at allA depth map plus a 2.5D projection gives a camera move on a still image with perfect consistency and zero generation cost, which beats a generated clip outright whenever the subject must stay exactly itself.
- 18Predicting which second will breakMost clip failures are visible in the still before any money is spent, because the same nine features — contact, thin structure, text, reflection, crowd, pattern, transparency, hair and count — fail for reasons you already know.
- 19Every still you keep is a part you do not have to make againA still with its prompt, seed, model version and provenance stored beside it is a reusable part, and a project without that record cannot be revised six weeks later — it can only be redone.
- 20Where this is already the right toolGenerative video is already the right tool for b-roll, impossible shots, pre-visualisation and short-form, and still the wrong tool wherever a specific person or object has to stay itself.
Module 3
Directing, not describing
The prompt is a light influence over a great deal of structure, so the words that steer are the ones a shot list uses — size, angle, lens, one move with a speed, a verb for the subject and a note on what stays still. This block builds that vocabulary properly, adds the controls that take motion out of language altogether, and finishes with an experimental protocol that tells you which of your instructions the model is actually following.
By the end you can
Write a shot instruction naming size, angle, lens character, one camera move with a speed, the subject's verb and what stays still; choose between language, a motion brush and an explicit conditioning signal for a given move; and run a fixed-seed experiment that determines which parts of your instruction the model obeyed
- 21Cinematic, 8k, epic gets you the average of everythingName the shot size, one camera move, the subject's verb and what stays still — mood adjectives return the average of everything captioned that way.
- 22The two words that decide most of the frameShot size and camera angle were named explicitly in the captions the model trained on, so they steer more reliably than any other words, and they carry meaning an audience reads without noticing.
- 23Focal length, depth of field, and the words that carry themFocal length changes the relationship between foreground and background rather than just the field of view, so naming a lens steers the geometry of the frame in a way that no amount of describing the subject can.
- 24Where people stand, and which way they faceScreen direction is a continuity contract across a cut, and because a generated shot has no set and no camera position you have to enforce it in the first frame rather than discovering it on the day.
- 25Taking motion out of languageLanguage is a poor way to specify a curve, so the controls that matter most are the ones that bypass it — a motion strength slider, a painted region, a drawn trajectory — and each of them fails in a way the prompt equivalent does not.
- 26Depth, pose and flow as instructionsA conditioning signal supplies the model with a per-frame structural target — depth, pose skeleton or optical flow — which is why it produces motion a prompt cannot describe, and why it also imports every flaw of the source video.
- 27What a negative prompt can and cannot removeA negative prompt is a second conditioning the model is pushed away from, which works well for qualities of rendering and badly for objects, because naming an object pulls it into the guided direction before the negation subtracts it.
- 28An experiment protocol that fits on a cardBecause generation is stochastic, comparing two clips that differ in more than one respect tells you nothing, so every useful conclusion about a model comes from a fixed seed, one changed variable and a written record.
- 29The clauses that quietly do nothingPrompt adherence degrades with the number of independent requirements, and the ones dropped first are counts, spatial relations, negations and anything about timing — because a few hundred text tokens are steering a quarter of a million video tokens.
Module 4
Making separate shots belong to each other
Nothing in a video model remembers your character, your room or your look between generations, so continuity is something you impose from outside. This block works through every mechanism available — a character sheet, reference conditioning, a trained identity model, reusable plates, a written look file, and repair after the fact — and finishes with the arithmetic nobody quotes: what a per-shot success rate does to the odds of getting eight matching shots in a row.
By the end you can
Hold a character, a location and a look across a multi-shot sequence using a character sheet, reference conditioning or a trained model as appropriate, decide which of those a given job justifies, and state the compounding probability of a matching sequence from a measured per-shot hit rate
- 30A cut breaks on the light before it breaks on the faceIdentity holds because the first frame holds it, and a cut breaks on mismatched light and wardrobe long before it breaks on the face.
- 31The document you build before you generate anythingA character sheet is a fixed set of images of one person under one light in one wardrobe, and it works because every later first frame is built from it rather than described afresh.
- 32What a reference image is actually doingReference features condition on an embedding of the supplied images rather than storing the person, so they reliably reproduce a type — build, colouring, wardrobe, general impression — while the exact facial geometry keeps re-sampling within that type.
- 33When a LoRA is worth the afternoonTraining a small adapter on twenty images teaches the base model one specific identity as a real concept, which is the strongest consistency tool available and only worth it when the character will appear across enough shots to repay a fixed setup cost.
- 34Building a room that stays the same roomA location is easier to hold than a face because a plate can be literally reused — generate the space once, then derive every shot in that scene from the same image by cropping, reframing and compositing.
- 35The blue that is not quite the same blueAudiences detect wardrobe and prop discontinuity faster than facial drift because clothing is a large area of uniform colour, and colour differences of a few per cent across a large area are exactly what human vision is built to notice.
- 36Fixing a face after the clip existsFace restoration and face replacement operate per frame on a finished clip, which makes them useful for small drift and dangerous for large changes — and the same tooling that rescues your own shot is the tooling behind non-consensual synthetic media.
- 37Deciding the look once, in writingA look is a set of decisions about colour, contrast, texture and lens that should be made once and written down, because a piece where every shot was styled individually reads as a collection rather than as a film.
- 38The arithmetic nobody puts in the demo reelPer-shot success rates multiply, so a hit rate that feels acceptable on one shot becomes a low probability across a sequence — which is why the professional response is to reduce the number of shots that must match rather than to improve the rate.
- 39Six things it cannot do, and the reason for eachEach failure has a mechanism — no simulation, no symbols, no persistent objects, no reference for your specific thing — and the mechanism tells you whether to wait for a better model or change the shot.
Module 5
The half of video that is not picture
Sound is the cheapest quality upgrade in the pipeline and the one most people skip. This block covers the talking head honestly — what lip-sync is actually reshaping and where it fails, what performance transfer captures that phonemes do not — and then the audio nobody sees: cloned and recorded voice, room tone, spot effects, music licensing you can defend, and why a well-chosen sound makes a viewer forgive physics the model got wrong.
By the end you can
Build the audio half of a generated video — a speaking character whose consent you can document, a voice recorded or synthesised with a stated quality trade-off, room tone and spot effects under every shot, and music whose licence you could show a platform — and explain why sound repairs perceived physical errors that no regeneration would fix
- 40Lip-sync is easy; the permission is notPermission to film is not permission to synthesise, and the only label that survives a screenshot is the one burned into the frame.
- 41What the model is actually reshapingAudio-driven lip-sync maps sound to a small set of visually distinct mouth shapes and inpaints the mouth region frame by frame, which is why it works at medium shots and falls apart on plosives, profiles and close-ups.
- 42Acting is timing, and timing has to come from somewherePerformance transfer drives a generated face from a recorded one, capturing the pauses, emphasis and eyebrow movement that carry meaning — which is why it beats lip-sync on anything with emotional content and why the person in front of the webcam matters more than the model.
- 43Synthetic voice, and where it stops being good enoughText-to-speech has solved pronunciation and timbre but not interpretation, so synthetic voice holds up for short informational lines and degrades over long-form narration where a listener starts noticing that nothing is being meant.
- 44A phone, a quiet room, and four minutes in AudacityMost of recorded voice quality comes from the room rather than the microphone, so a phone in soft furnishings beats an expensive microphone in a hard empty room, and a fixed four-step cleanup chain handles the rest.
- 45Why silence is the worst soundGenerated clips arrive with no sound at all, and true digital silence reads as broken, so a continuous ambience bed under every shot is the single cheapest thing that makes generated footage read as filmed.
- 46Music you could defend if somebody askedMusic is the easiest part of a soundtrack to obtain and the easiest to get legally wrong, because a licence has to cover the specific use, and AI-generated music sits on unresolved ground about ownership and training data.
- 47The viewer believes the soundWhen sound and picture disagree, perception leans on the sound, which is why a well-synced impact makes a physically wrong collision read as correct and why sound design is a repair tool as well as a finishing one.
- 48When the model makes the sound tooNative audio generation produces synchronised ambience and dialogue in one pass, which is remarkable and usually unusable as a final track, because you cannot separate, replace or adjust anything in it afterwards.
Module 6
From generations to a finished piece
Most of the difference between footage and output happens after the last generation. This block is the post-production half in order: conforming mixed frame rates onto one timeline, selecting and assembling, hiding cuts where the eye is already moving, pulling thirty independent colour responses into one world, laying grain that replaces the texture the encoder removed, adding real text, retiming without exposing fake physics, and exporting to a spec that survives a platform's re-encode — including on a machine that cannot run a professional editor.
By the end you can
Take a folder of mixed-rate generations to a delivered file — conformed, assembled, cut on motion, colour-matched under one grade and one grain pass, titled, sound-finished and exported to a named platform specification — and name the free tool that performs each step on the hardware you actually have
- 49The generator makes footage; the editor makes the videoTrim the settle and the drift, unify everything with one grade and one grain pass, and put real sound underneath — that is most of the difference between footage and output.
- 50Decide the frame rate before you import anythingA timeline has one frame rate and one resolution, and every clip that does not match is resampled to fit — so the settings you choose before importing determine whether your footage judders for the rest of the project.
- 51Choosing takes before you start cuttingSelecting takes and assembling a rough cut are separate passes with different judgements, and collapsing them is why edits stall — you cannot decide whether a shot works in the sequence while you are still deciding whether the shot works.
- 52Cut on motion, and one frame before the failureA cut placed while the eye is already tracking change is far less visible than one on a static frame, which gives you both a craft rule and a way to place every cut exactly where a generation stops working.
- 53Thirty renders, one worldEach generation makes its own independent decisions about colour and contrast, so a piece is unified by a two-stage process — shot matching to bring every clip to a neutral common ground, then one grade over all of it — and never by grading each clip to taste.
- 54Putting back the texture the encoder removedGenerated frames are smooth because the autoencoder discarded high-frequency detail, so a single grain pass across the finished timeline replaces a missing surface property and unifies clips more effectively than any colour operation.
- 55Real type, added afterwardsText belongs in the editor rather than in the generation, because a title is vector type that stays identical for every frame while generated letterforms are reinvented each frame from a latent that never held them.
- 56Slow motion is where fake physics goes to be caughtSlowing generated footage gives the viewer time to verify physics the model only approximated, so retiming should be used to speed things up and to find a moment, and almost never to dwell on a physical event.
- 57The last render, and what the platform does to itEvery platform re-encodes what you upload, so your export should be a high-bitrate master that gives their encoder good material to work from rather than a small file that has already thrown away the detail.
- 58The whole pipeline on a phone or an old laptopEvery step of post-production has a version that runs on modest hardware, and the constraint that actually bites is playback performance rather than capability — which proxies solve for free.
Module 7
Money, clients and running the job
Generative video is the first creative tool in years with a meter running, which changes how a project has to be planned, quoted and executed. This block builds the production apparatus: a shot list that can be batched, an animatic that gets approval before anything expensive happens, a draft-then-final discipline, queue management through an API, the arithmetic of subscription against top-up, a quote priced in shots with a stated attempt allowance, and the record that makes a revision six weeks later a small job rather than a reshoot.
By the end you can
Plan, price and run a generative video job end to end — a shot list with durations and dependencies, an approved animatic before any final generation, a draft pass at the cheapest tier, a kill number per shot, a quote expressed in shots with included attempts, and a project record that makes a later revision cheap
- 59An afternoon of curiosity can cost more than a day of workPrice your work in shots with a fixed number of included attempts, because the cost driver is generations thrown away, not hours spent.
- 60The document everything else is built fromA shot list with a duration on every line converts an open-ended creative task into a countable one, which is what makes batching, quoting and stopping possible.
- 61Get the yes before you spend the moneyAn animatic made from stills costs almost nothing and carries the same approval as a finished cut, which moves the expensive client conversation to before generation rather than after it.
- 62Test the idea at a tenth of the priceA draft generation at low resolution on a distilled model answers whether a motion idea works, which is a different question from whether the shot is good, and answering it cheaply first is where most of the saving in a project lives.
- 63Queue everything, then judge it togetherGenerating one clip at a time invites the just-one-more loop that destroys budgets, while queueing a batch and reviewing the results together produces better decisions and a countable spend.
- 64The arithmetic nobody does before subscribingCredits usually expire at the end of a billing period, so a subscription is cheaper per second only if you actually finish a project inside that window and is worse than useless in a month where you do not work.
- 65Price the thing that actually costs moneyThe cost driver in generative video is generations discarded rather than hours worked, so a quote expressed in shots with a stated attempt allowance puts the real variable where the client can see it and decide about it.
- 66What to keep, and the day it saves youA project record is what turns a later revision into a twenty-minute job instead of a reshoot, and it has to be assembled during production because most of it cannot be reconstructed afterwards.