Fine-Tuning, and When Not To
The case against fine-tuning first, then how to do it properly.
- Level
- Assumes you have built something
- Lessons
- 84
- Reading time
- 758 min
- Price
- Free, no sign-up to read
Fine-tuning is the most over-reached tool in applied machine learning. This course opens with the honest answer — a better prompt or retrieval solves most of what people want fine-tuning for — and then, for the cases that survive that test, teaches the practice end to end: what training actually changes and what it cannot add, LoRA and QLoRA memory arithmetic you can do on paper, building a dataset and the held-out set that has to come before it, chat templates and loss masking, running a job on a free T4 or a rented card for pennies, catching catastrophic forgetting, measuring whether it helped, and serving the adapter. Written for a mid-range laptop as readily as a rented A100.
Opens after the Running Models Yourself exam
Sign in, finish that course, and pass its exam. You can read this syllabus meanwhile.
Go to Running Models YourselfModule 1
Deciding whether to fine-tune at all
Most fine-tuning projects should not happen. This module gives you the diagnosis that tells you which failures weights can fix, the cheaper levers that fix the rest, and the true cost of owning a model you trained yourself.
By the end you can
Diagnose a model failure as a format, knowledge, style, capability or cost problem, name the cheapest lever that fixes that class, and defend in writing a decision to fine-tune — or not to — against the prompt, retrieval, encoder and distillation alternatives and their licence constraints
- 1The Honest Answer FirstA prompt changes in ten seconds; a fine-tune changes in a day. Exhaust the fast loop first.
- 2What Fine-Tuning Changes, and What It Cannot AddFine-tuning teaches the shape of an answer, not its content; facts belong in the context window.
- 3Diagnosing the failure before choosing the fixSort thirty real failures into format, knowledge, style, capability, adherence and cost before choosing a fix; only adherence, style and cost have fine-tuning as their cheapest answer.
- 4Try the retriever firstRecall@k is a hard ceiling on a RAG system's accuracy, so measure it before blaming the generator — and when something must be trained, the retriever is a hundred times smaller and outlives the base model.
- 5When the answer is one of six labelsWhen the output is one of a fixed set of labels, a 150M encoder fine-tuned on a thousand examples usually beats prompting a large decoder on accuracy, latency and cost — at the price of needing labels and retraining to add a class.
- 6Distillation, the case that usually survivesDistillation is the fine-tuning case with no cheaper rival, but the student is capped by the teacher on the traffic you sampled, so the filter between them — verifiers, rejection sampling, a hundred pairs read by a person — is where the quality actually comes from.
- 7The cost of owning a modelThe training run is the cheapest part of a fine-tune; the real cost is a dataset, an evaluation set, a serving line item and a rebuild every time the base model is superseded, so the thing you must be able to reproduce is the pipeline, not the weights.
- 8Licences, and what you may train onModel licence, data licence and downstream obligations are three separate permissions, the underlying copyright question is genuinely unsettled and differs by country, and per-example provenance recorded at collection time is the only thing that makes the question answerable later.
- 9Writing the decision downOne page written before any training — quantified failure, diagnosis, frozen baseline, budget with a kill criterion, a list of what must not get worse, and a rebuild estimate — is what makes a fine-tune's result believable or its abandonment cheap.
Module 2
How training actually works
One optimisation step, taken apart: the forward pass, the loss, the gradient, the optimiser state, and the precision the numbers are stored in. Every hyperparameter you will later set moves one of these.
By the end you can
Trace one optimisation step from a batch of tokens to a changed weight, and predict which way loss, memory and stability move when you change the learning rate, the batch size, the precision or the number of epochs
- 10One training step, taken apartA training step is forward, cross-entropy against the next token, backward to gradients, optimiser update — and the memory, the masking and every hyperparameter you will set are consequences of those four lines.
- 11Reading the loss numberCross-entropy is readable as a probability — exp(-loss) is the average chance given to the right token — but it is comparable only against your own runs, and because it is a proxy you must select checkpoints on a task metric, since validation loss can rise while the metric you care about improves.
- 12Optimisers, and what their state costsAdamW's two per-parameter moment buffers cost 8 bytes per trainable parameter, which is why full fine-tuning is memory-bound and why LoRA makes the optimiser irrelevant; 8-bit and paged variants buy that memory back when you must train everything.
- 13Learning rate, warmup and the scheduleLoRA needs a learning rate ten to twenty times higher than full fine-tuning because its adapter starts at exactly zero, warmup exists to stop AdamW taking a large step from a one-sample variance estimate, and a cosine schedule must be set for the run you actually intend to finish.
- 14Batch size, accumulation and the token countEffective batch is per-device times accumulation times devices, the unit that actually governs gradient noise is tokens per step rather than examples, and loss must be normalised by real token counts across an accumulation window or long and short examples get equal weight.
- 15Precision, and why bf16 ended a class of bugsbf16 keeps fp32's exponent range and sacrifices mantissa bits, which removes the underflow that forced fp16 to use loss scaling — and since the free T4 has no bf16, free-tier runs still live with the scaler and its skipped steps.
- 16Where the memory actually goesMemory is weights, gradients, optimiser state, activations and logits — and at long context the quadratic attention matrix and the vocabulary-sized logits tensor are usually the two surprises, both removable with flash attention and a fused cross-entropy before you touch batch size.
- 17Epochs, overfitting and checkpoint selectionThe step down in training loss at each epoch boundary is memorisation, not learning; select checkpoints on a task metric with a canary set beside it rather than on validation loss, because the best checkpoint is often the last one before general capability starts to slip.
- 18Regularisation that helps, and the kind that does notFewer epochs, more data and a replay mixture regularise a fine-tune far more than any config knob, gradient clipping at 1.0 is cheap insurance against one pathological example, and label smoothing flattens the output distribution in a way that harms generation even though it helps classifiers.
Module 3
Full, LoRA and the parameter-efficient family
The methods that made fine-tuning affordable, with the arithmetic to predict memory before you rent anything, and an honest account of which variants are worth their extra complexity.
By the end you can
Predict on paper the memory a given model, method and sequence length will need, choose between full fine-tuning, LoRA, QLoRA and the tiny methods for a stated budget, and justify a rank, an alpha and a target-module set
- 19Full, LoRA, QLoRA, and the Memory ArithmeticFull fine-tuning costs about 16 bytes per parameter; LoRA charges that only on the 1% you train.
- 20What LoRA is actually doingLoRA freezes W and learns a rank-r factorisation of the correction, initialised so the adapter starts as an exact no-op — which is why it needs a much higher learning rate, why it cannot be worse than the base at step one, and why it adds reweighting rather than new machinery.
- 21Choosing rank, alpha and which modules to touchAdapting all linear projections at r=16 beats adapting the attention pair at a high rank, alpha should be fixed at 2r so it does not confound your learning-rate sweep, and target_modules="all-linear" survives a change of model family where a copied name list does not.
- 22QLoRA: quantising the part you are not trainingQLoRA quantises only the frozen base to NF4 with block-wise scales and double-quantised constants, buying a 4x memory reduction for roughly 30% throughput and an adapter that was fitted against a slightly wrong model — which is why merging it back is the awkward part.
- 23DoRA, rsLoRA and the variant questionrsLoRA and DoRA fix specific defects — the alpha-over-r damping at high rank and the coupling of magnitude to direction at low rank — but variants are the last term in the sum, behind the dataset, the template, the learning rate and the target modules.
- 24Soft prompts and prefixesSoft prompts and prefixes train a few hundred kilobytes of learned vectors with every weight frozen, which makes per-tenant adaptation almost free to store — but they steer the computation rather than changing it, so they close the gap to full fine-tuning only at large model scale.
- 25The tiny methods, and a gotcha in one of themIA³ adapts a model with three learned scaling vectors per layer, about 0.01% of parameters, merging exactly into the weights and suiting tiny datasets — while BitFit is nearly a no-op on Llama-style models because they have almost no bias terms to train.
- 26Full fine-tuning, when it is genuinely rightFull fine-tuning is right for a new language, a very large dataset, a small model or a distillation student, costs about 16 bytes per parameter in persistent state before activations, and forgets precisely because every weight is free to move — so run the LoRA first to find out whether the data is worth it.
- 27Training on a free or nearly free budgetA free Kaggle T4 with 12-hour sessions trains an 8B QLoRA at sequence 1,024, Unsloth's rewritten kernels roughly double that throughput, and the whole chain from dataset to a GGUF model running on your own laptop costs nothing — with checkpointing to Drive or the Hub being what makes a killable session survivable.
Module 4
Building the dataset
The dataset is the model. This module covers where examples legitimately come from, the formats they must be converted into, the filters that decide quality, and the record that lets somebody else rebuild the set.
By the end you can
Assemble a training set from named sources with checked licences, convert it to one chat format, filter and deduplicate it, mix in replay data at a stated ratio, and publish a data card that lets a stranger reproduce the set
- 28The Dataset Decides EverythingYour model learns your dataset's whole distribution, including the habits you never meant to teach.
- 29Where the data legitimately comes fromYour own traffic has the right distribution, hand-written examples cover what traffic never contains, and public sets are for replay — with a dataset's own licence and the terms under which its contents were generated being two separate permissions that a provenance column recorded at collection time is the only way to answer later.
- 30The three formats, and the conversion trapConvert everything to messages format once on the way in, use the same formatting function for training and inference, and print one templated example as a string before every run — because the commonest data bugs are a changed separator, a half-present system prompt, and truncation that removes the end-of-turn token.
- 31Cleaning at scaleFilter in order of cost — exact hashes, then MinHash near-duplicates, then length, truncation, language and repetition heuristics — and then read a hundred examples by hand, because the systematic annotation problems that decide a fine-tune's quality have no automated detector.
- 32Writing the answers, and keeping them consistentThe model learns the distribution of your answers rather than their correctness, so unforced variation between annotators becomes randomness in production — which makes a one-page style guide with exact wording for the declining case worth more than another thousand examples.
- 33Generating a dataset from a teacherDraw the inputs from real traffic and only the outputs from the teacher, sample several completions and filter them with a verifier wherever correctness is checkable, then strip the teacher's stock phrases and check that your generated set covers your traffic's distinct intents rather than just its volume.
- 34Mixtures and replayMixing 15 to 30% general instruction data into a task-specific set preserves general capability better than any training-side mitigation, because the weights supporting that capability keep receiving a signal to stay put — and shuffling the file is not optional when the learning rate decays.
- 35Packing, document boundaries and long contextPacking removes padding but lets one document attend to the previous one unless the block-diagonal mask and per-document position ids are actually in the batch — and extending context requires both a RoPE scaling change and long training data that, for most tasks, does not naturally exist.
- 36Multi-turn conversations and tool callsCompute loss on every assistant turn including the tool-call message, mask the tool's response because it is the world's output rather than the model's, and include conversations where no tool was needed — a dataset of only successful calls produces a model that always calls something.
- 37Memorisation, and what you cannot take backMemorisation rises sharply with how often a sequence is duplicated, which makes deduplication a privacy control as much as a quality filter — and since you cannot reliably remove one person's influence from trained weights, the engineering answer is a dataset you can rebuild and a recorded link between data version and model.
- 38The data cardA data card with per-filter counts, an honest known-gaps section, a content hash and a one-command rebuild is what links a model to the data that made it — and writing the gaps section usually changes the dataset before anybody else reads it.
Module 5
Objectives beyond next-token prediction
Supervised fine-tuning teaches a model to imitate. Preference methods teach it what to prefer, and verifiable rewards teach it what is correct. Each needs different data and is gamed in a different way.
By the end you can
Choose between supervised fine-tuning, a preference method such as DPO or KTO, a verifiable-reward method such as GRPO, and continued pretraining, and state the data each needs, the memory each costs, and the way each is gamed
- 39What supervised fine-tuning cannot expressSFT can only say 'make this more likely', so it cannot express that one answer is better than another or correct a failure it never sees — which is why the method you need is decided by whether you can write the right answer, only rank two answers, or check one with a program.
- 40Where preference data comes fromPreference pairs must be on-policy — the rejected side generated by the model you are about to train — or you are teaching it to avoid behaviour it never had; and pairs where you cannot state in one clause why the chosen answer is better are noise that trains something arbitrary.
- 41DPO, from first principlesDPO replaces the reward model with an implicit reward expressed as the log-probability ratio against a frozen reference, needs no second model copy when you use LoRA, and fails in a characteristic way — both chosen and rejected probabilities falling — which is why rewards/chosen is the statistic to watch.
- 42KTO, ORPO, SimPO and choosing among themChoose among the preference methods by what data you hold — KTO for unpaired thumbs, ORPO to merge SFT and preference into one reference-free stage, SimPO when length is inflating, IPO for noiseless constructed pairs — because their benchmark differences are smaller than the differences between datasets.
- 43RLHF and PPO, honestlyRLHF's PPO stage holds four networks — policy, frozen reference, reward model and value critic — inside a generation loop, which is why DPO replaced it for most applied work; but its objective is the one DPO was derived from, and its KL-versus-reward plot is the clearest picture of a policy exploiting its reward model.
- 44Verifiable rewards and GRPOGRPO drops PPO's value model by using a group of sampled responses as its own baseline, which makes verifiable-reward training affordable — but every group that scores identically produces no gradient, and rejection-sampling fine-tuning on verified outputs is the cheaper baseline you should beat first.
- 45Reward hackingOptimisation pressure finds the gap between your reward and your intent, so assume yours is exploitable and instrument for it — a held-out reward you never train against, logged proxy statistics like mean length, and twenty outputs read by eye are what make the hack visible.
- 46Continued pretraining on raw textContinued pretraining puts loss on every token of raw text to install a domain's register and vocabulary, needs hundreds of millions of tokens rather than thousands of examples, and reliably removes instruction-following — so it is always followed by redoing SFT, and it still will not make the model reliable about any specific document.
- 47Training a head instead of generating tokensSwapping the token head for a linear head turns a decoder into a scorer that costs one forward pass instead of a generation loop, but a preference-trained reward model learns an ordering with no absolute scale, and a decoder classifier pools the last non-padding token — so padding side and pad_token_id must both be set.
Module 6
The mechanics of a run
The part where things silently break: tokenizers, special tokens, masking, seeds, checkpoints and multi-GPU. Most failed fine-tunes are not bad ideas, they are configuration bugs nobody decoded.
By the end you can
Configure and launch a fine-tuning job with the correct chat template, masking and tokenizer handling, and diagnose from the logs alone whether a run is not learning, diverging, or memorising
- 48Chat Templates and Loss MaskingDecode one batch's unmasked labels before every run; it catches most fine-tuning bugs in five seconds.
- 49Running a LoRA, and What It CostsGPU time is the cheap part: a 7B LoRA run costs cents, while the dataset and the evaluation cost days.
- 50The tokenizer is part of the modelThe tokenizer is a fixed part of the model whose efficiency on your language decides cost, context and quality — and setting the pad token equal to the end-of-sequence token, then masking by id, removes the gradient that teaches the model to stop.
- 51Adding tokens, and initialising them properlyNew vocabulary rows are initialised randomly by default, which places them outside the region existing embeddings occupy — so set them to the mean of existing embeddings or of the tokens they stand for, and remember they stay frozen under LoRA unless embed_tokens and lm_head are in modules_to_save.
- 52A working config, line by lineA working config is a set of decisions, not defaults — the instruct variant as base, all-linear targets, 2e-4 for LoRA, non-reentrant checkpointing so gradients reach the adapter, and length-grouped batching instead of packing until packing is verified.
- 53Debugging a runDecode the unmasked labels of one real batch before every run — it should print exactly the assistant's answer and nothing else — and if a pipeline cannot memorise eight examples in a hundred steps, the problem is the code rather than the data.
- 54What to log, and how to read itGradient norm, tokens per step, token accuracy and three generated samples tell you things loss cannot — a run heading for divergence is visible in the gradient norm hundreds of steps before the loss moves, and stylistic collapse is visible only in the samples.
- 55Checkpointing and resumingA resumable checkpoint holds optimiser state, scheduler position and RNG state as well as weights, so resuming is only correct when the data order and the number of devices are unchanged — and on interruptible instances the checkpoint must live somewhere the instance does not.
- 56Reproducibility, and the noise floorSet a seed and record the git commit, config hash, dataset hash, base revision and library versions — then run three seeds to find your noise floor, because an improvement smaller than twice that spread is a coin flip you have named a decision.
- 57Multi-GPU without tearsAdding GPUs multiplies the effective batch size by the device count, so gradient accumulation must be divided or the learning rate raised — and sharded strategies over PCIe often give back in communication what they gain in parallelism, which is why a single card is the faster path for any run under about twelve hours.
Module 7
Fine-tuning beyond a chat model
Embedding models, rerankers, classifiers, speech, vision-language and image generators. Smaller models, smaller datasets, larger wins — and in several of these the cheaper technique beats fine-tuning outright.
By the end you can
Adapt an embedding model, a reranker, a text classifier, a speech model, a vision-language model or an image generator, and name in each case what differs from fine-tuning a text LM and which cheaper technique should be tried first
- 58Fine-tuning an embedding modelEmbedding models train on query-passage pairs with in-batch negatives, which makes batch size the real hyperparameter — gradient caching buys a 512-wide batch on a small card — and mined hard negatives must be taken from below the top ranks, because the top ones are often correct answers you would be teaching the model to reject.
- 59Training a rerankerA cross-encoder reads query and passage together so it ranks far better than a bi-encoder, but it must run once per candidate — which is why it reranks 50 results rather than searching a corpus, and why the pretrained one should be measured before you train your own.
- 60Fine-tuning a classifier properlyA 150M encoder with id2label in its config, a weighted loss or adjusted thresholds for imbalance, macro F1 with the confusion matrix read rather than the summary line, and a four-point learning curve that tells you whether another week of labelling is worth it.
- 61Fine-tuning a speech modelTry an initial prompt with your domain vocabulary and a larger checkpoint before training, resample to 16 kHz mono, freeze the audio encoder on small datasets, and report raw as well as normalised word error rate alongside recall on the terms that actually matter.
- 62Fine-tuning a vision-language modelFreeze the vision encoder, train the projector and LoRA the language model, and check how many sequence positions one image actually consumes before planning memory — and for printed documents, an OCR pipeline into a text model is usually cheaper, more accurate and far easier to debug.
- 63LoRA for image models, and the likeness questionImage LoRAs train on 10 to 30 images in under an hour, and what you caption becomes controllable while what you leave uncaptioned is baked into the subject — while likeness training is technically trivial and legally unsettled in ways that differ by jurisdiction, which makes consent and labelling the defensible position rather than a legal conclusion.
- 64Getting structure without training for itConstrained decoding masks illegal tokens at every step so invalid output cannot be produced, which makes it a free floor under any structured task — it guarantees syntax and nothing about truth, so keep it even after fine-tuning rather than treating the two as alternatives.
- 65Small models, where fine-tuning matters mostFine-tuning helps more the smaller the model, because a small model has to be given a behaviour rather than prompted into one — and the case for on-device models is privacy, offline operation and zero marginal cost rather than price, against the specific cost that they collapse rather than degrade off-distribution.
- 66Tabular and time series: the honest answerGradient-boosted trees remain the default for predicting from tables, so language models belong around the problem — turning free-text columns into features, resolving entities — rather than in it, and any time-series foundation model has to beat seasonal naive on your own series before its benchmark numbers mean anything.
Module 8
Did it actually work
A fine-tune that improves your target metric and quietly breaks three other things is a loss. This module measures both halves, with intervals, on a set the model has not seen.
By the end you can
Compare a tuned model against a cost-matched baseline on a leak-free held-out set, report the difference with an interval, and demonstrate whether general capability, safety behaviour or calibration regressed
- 67Did It Help, and What Did It BreakA win rate without an interval and a canary set is not evidence, it is a feeling.
- 68The baseline you must beatCompare against three baselines — the frozen best prompt, the cost-matched alternative, and production today — on one table of quality and cost per thousand requests, because the alternative nobody was ever going to choose is not a baseline and a one-point difference on 200 items is not a difference.
- 69A held-out set that actually holdsRandom splits leak through near-duplicates, groups, time and templates, so split by group and by time and then measure cross-split near-duplicate overlap — and keep a locked set you open a handful of times, because an evaluation set you have consulted forty times has become a training set for your own decisions.
- 70Pairwise human comparison, done properlyBlind, order-randomised, tie-allowing comparisons with a stated criterion and a measured judge-agreement rate — with the number of comparisons fixed in advance from the effect size you would act on, and mean response length reported beside the win rate because length buys wins it did not earn.
- 71Judging a model you distilled from the judgeA teacher judging its own student rewards resemblance rather than quality, so use a verifier, a human sample or a judge from a different family — and a student that beats its teacher on teacher-judged comparison is a finding about the measurement, not the model.
- 72Contamination you can check, and contamination you cannotThirteen-gram overlap will find contamination between your training and evaluation sets, and the answer-ordering test can detect memorised benchmarks — but the base model's pretraining corpus is unchecked and unknowable, which is why a held-out set built after the training cutoff is the only evidence that holds.
- 73Safety regression, including from benign dataFine-tuning on entirely benign data measurably degrades refusal behaviour, because alignment is a thin late layer and every training example demonstrates compliance — so include refusal examples, measure with your own refusal set at each checkpoint, and never let the model's own alignment be the only safety control.
- 74Calibration after tuningFine-tuning sharpens the output distribution and preference tuning rewards confidence, so a tuned model's 0.9 no longer means 0.9 — which breaks every threshold, fallback and abstention rule downstream until temperature scaling is refitted on the exact artefact you serve.
- 75Ablations that earn their costRun an ablation only when its result would change a stated decision, and read every result against a three-seed noise floor — the data learning curve is the one that settles whether another week of labelling is worth it, and a written log of non-findings prevents the same experiment being run twice.
Module 9
Shipping it, and living with it
Serving, quantising, monitoring, versioning and the day the base model is replaced. A fine-tune is not a deliverable, it is a dependency you now maintain.
By the end you can
Serve a tuned model or adapter at a stated cost and latency, monitor it for drift, reproduce a six-month-old artefact from its recorded version, and plan for the day the base model is replaced
- 76Serving the AdapterMerging, quantizing and switching engines all change the model; evaluate the exact artifact you serve.
- 77Quantising the model you shipServe with AWQ on a GPU or Q4_K_M in GGUF on a laptop, merge into the full-precision base before quantising, calibrate on your own data — and run a three-way base, tuned, tuned-and-quantised comparison, because quantisation error can be as large as your adapter's delta and round the entire fine-tune away.
- 78Serving many adapters from one baseOne base plus fifty adapters is 24 GB where fifty merged models are 800 GB, at 10 to 20% throughput cost and a shared rank ceiling you must choose at training time — but one adapter trained on everyone's data with the customer named in the prompt often matches per-customer adapters, so check whether you are paying for quality or for isolation.
- 79The cost at steady stateAt two million requests a month a self-hosted tuned 8B on redundant cards costs more than a hosted small model before anybody is paid to maintain it — so the crossover sits above roughly two million requests or wherever latency, privacy or vendor stability overrides the arithmetic entirely.
- 80Monitoring a model that fails silentlyA drifted model returns a confident wrong answer with a 200 status, so monitoring means unlabelled proxies — embedding distance from the training distribution, response length, refusal and fallback rates, and above all the rate at which users rephrase — plus a weekly shadow run of the frozen set and every incident turned into a permanent evaluation item.
- 81The day the base model movesAn adapter fitted against one set of base weights loads cleanly onto a newer release and is silently worse, so pin the revision hash — and when a better base ships, first check whether it plus a good prompt already beats your tuned old model, because deleting the fine-tune is a success rather than a loss.
- 82Versioning so a model can be explained laterA manifest of content hashes travelling with the artefact, plus a model-version field on every logged request, is what turns 'what produced this answer in March' from a memory into a lookup — and rebuilding a random old version once a quarter is how you find out whether it actually works.
- 83Model cards and disclosureA model card states intended use, out-of-scope use, training-data provenance, evaluation with intervals, and limitations explained by mechanism — and since the regulatory position differs by jurisdiction and is still moving, that document is what travels everywhere while the specific legal duties do not.
- 84Retiring a fine-tuneSchedule the quarterly question of whether a fine-tune still beats its cost-matched baseline, retire it happily when the base model catches up, keep the decision record and cards after the weights are deleted — and treat a model nobody can rebuild as one to retire even while it is performing.