Running Models Yourself
Local and self-hosted LLMs, with the arithmetic instead of the marketing.
- Level
- Some background helps
- Lessons
- 82
- Reading time
- 717 min
- Price
- Free, no sign-up to read
A practical course on running large language models on your own hardware. It teaches the arithmetic that decides whether a model fits, what quantisation actually costs you, which serving engine wins under which conditions, and when local is genuinely cheaper than an API — with real commands, real numbers, and no engine advertising. Written for people on ordinary laptops as much as for people with a 4090.
Opens after the The Open Model Ecosystem exam
Sign in, finish that course, and pass its exam. You can read this syllabus meanwhile.
Go to The Open Model EcosystemModule 1
Deciding to run it yourself
Before any download, two questions have to be separated: is this model good enough for the work, and can this machine run it. This block answers the first one. What open weights are and what their licences actually permit, how to read a model card without being sold to, what each size class can genuinely do, why a mixture-of-experts model breaks the usual size intuition, and an hour-long audition that costs about a dollar and settles the question before you spend anything on hardware.
By the end you can
Decide whether a stated task belongs on your own hardware — by auditioning the exact open weights through a hosted copy, naming the constraint that forces local rather than assuming one, reading a licence well enough to say what you may ship, and stating the capability ceiling the weights impose regardless of the machine
- 1Why run a model yourself, and when not toThe weights set the quality ceiling; hardware only decides whether you can reach it and how fast.
- 2What "open" means, and what it does not"Open" splits into open weights, open source and open data — most famous models are only the first, and a base model's licence and use policy follow every fine-tune and quant made from it.
- 3Reading a model card without being sold toThe Files tab of a model card is verifiable and the prose is a claim — config.json, the shard sizes and the chat template tell you more in three minutes than any benchmark table on the page.
- 4What each size class can actually doChoose a size class by the job's sensitivity to size — classification and extraction barely care, multi-step reasoning and tool loops care enormously — and try a newer or task-specialised model in the same class before reaching for a bigger one.
- 5Mixture of experts, and why it changes the sumA mixture-of-experts model costs memory like its total parameter count and generates at the speed of its active parameter count, which makes it the right choice on a machine with abundant slow RAM and the wrong one on a small GPU.
- 6Base, instruct and reasoning: three different productsBase, instruct and reasoning variants are different products from the same architecture — check for a chat template before blaming the model, and never feed a thinking block back into the next turn.
- 7The audition: settling it for about a dollarTest the weights before the hardware — thirty real items graded blind against a hosted copy of the exact open model answers the question a GPU purchase cannot, for under a dollar.
- 8The ceiling the weights imposeEvery capability limit of a local model lives in the weights, so design around it — retrieve rather than recall, one step per call, constrain and verify the output, and route the hard minority elsewhere.
- 9Local is not automatically privateLocal weights do not leak, but the application, the plugins, the hybrid features and above all your own prompt logs can — verify with a firewall block and a traffic capture rather than trusting the word local.
Module 2
The memory arithmetic
Everything about running a model on your own machine reduces to two sums and one ceiling: what occupies memory, and how fast memory can be read. This block does both properly — the KV cache and why two models of the same size can differ sixfold, the two very different phases of inference and why only one of them is helped by a faster processor, where the weights physically live and what each hop costs, what a second graphics card actually buys, and the ordered list of things to try when it does not fit.
By the end you can
Compute, for any model and any context length, whether it fits on a stated machine and roughly how fast it will run — naming which of weights, KV cache, bandwidth or interconnect is the binding constraint, and choosing the remedy that costs the least quality when it does not fit
- 10Does it fit: the arithmeticWeights are fixed; the KV cache grows with context and layers, and that is what usually breaks the budget.
- 11What actually runs on 8GBGeneration speed is memory bandwidth divided by model size; cores and capacity do not change that ceiling.
- 12The KV cache, in depthThe KV cache is the term that decides whether a model fits, and its size per token is set by the attention architecture — grouped-query, sliding-window and latent attention differ by several times at the same parameter count.
- 13Prefill and decode are two different machinesPrefill is compute-bound and decode is memory-bandwidth-bound, so the same hardware change can transform one and do nothing for the other — measure and reason about them separately, always.
- 14Where the model physically livesSpeed is decided by which memory the weights sit in, and a partial offload behaves like a harmonic mean — a tenth of the layers left on the CPU can cost half the speed, so fitting entirely usually beats fitting better.
- 15CPU inference, honestlyCPU generation speed is set by memory channels and memory speed rather than cores — populate every channel, thread to physical cores, and expect cores to help prompt processing while doing almost nothing for generation.
- 16More than one GPUA second card doubles memory almost for free with a layer split, speeds one user only under tensor parallelism and only over a fast link, and often serves concurrent users best as two independent copies of the model.
- 17Measuring what you actually haveTen minutes of measurement on your own machine — prefill and decode separately, memory used, temperature over time — replaces every published benchmark and diagnoses most local performance problems immediately.
- 18When it does not fit: the ordered remediesWork out whether weights or cache is the oversized term, then walk the ladder from free changes — context length, 8-bit cache, evicting other processes — down to quantisation and offload, which cost real quality.
Module 3
Quantisation and model formats
Almost nobody runs a model at the precision it was trained in, so almost everybody is running a lossy copy and most do not know what it cost them. This block opens the file: what a GGUF actually contains, how k-quants and importance matrices decide which weights to protect, what the GPU-side formats do differently, how to quantise a model yourself, and how to measure the damage instead of quoting a perplexity number that cannot detect it.
By the end you can
Choose a quantisation format and bit rate for a stated model, machine and task; produce a quantised model yourself from the original weights; and measure the resulting damage with a method that can actually detect it, rather than relying on a perplexity delta that cannot
- 19Quantisation, properlyQuantisation damages precision-critical work first — syntax, long instructions, rare tokens — while fluency stays intact.
- 20Inside a GGUF fileA GGUF is one memory-mappable file carrying weights, tokeniser and chat template together, with different tensors quantised at different bit rates — and its metadata can be inspected and repaired without re-quantising anything.
- 21K-quants and importance matricesK-quants spend bits where the model is sensitive and importance matrices decide where that is — so the calibration text silently determines which languages and tasks a low-bit quant preserves.
- 22GPTQ, AWQ and the GPU-side formatsGPTQ compensates for rounding error as it goes and AWQ rescales the few channels that see large activations, but the kernel that reads the file matters as much as the algorithm that wrote it.
- 23FP8 and models born quantisedFP8 spends its bits on exponent rather than precision, which is why it survives the outliers that hurt integer quantisation — and a quantisation-aware-trained model at the same bit rate beats any post-training quant of it.
- 24Quantising a model yourselfQuantise from the original weights, never from another quant, and check the chat template and tokeniser survived the conversion — those two mistakes account for most locally produced models that appear to be broken.
- 25Measuring the damage, properlyPerplexity averages away exactly the tokens quantisation damages — measure with KL divergence against the original and with a task set containing exact-syntax, long-instruction and non-English items at temperature 0.
- 26Bigger and crushed, or smaller and cleanFor a fixed memory budget, more parameters at lower precision wins down to about 4 bits — but the rule optimises quality per gigabyte, and speed, context length and task specialisation routinely override it.
- 27The other ways to make a model smallerQuantisation, pruning and distillation each remove a different kind of redundancy, and knowing which one a released model used predicts how it will fail — pruned models unevenly, distilled models confidently, quantised models on precision.
Module 4
Getting it running
The gap between a downloaded file and a working assistant is a handful of unglamorous details, and each of them has a failure mode that looks like a stupid model. This block covers the whole path: choosing and installing the right build for your hardware, fetching weights without wasting a day of bandwidth, the chat template that decides whether the model understands you at all, context settings that truncate in silence, the API every engine speaks, constrained output that parses, and the embedding, speech and vision models that run beside the language model.
By the end you can
Take a model from a repository to a working local endpoint on your own hardware — correct build, correct template, correct context configuration — call it from code through the OpenAI-compatible API with structured output that validates, and add embedding, transcription or vision models alongside it
- 28Ollama and llama.cppOllama is llama.cpp plus a registry, a template and sane defaults — leave it only for a flag it does not expose.
- 29Interfaces: what each one is forThe interface never changes the model, only what gets sent — so compare requests, not apps.
- 30Installing the right build for your hardwareThe build has to match the accelerator, and a binary compiled without your backend runs silently on the CPU — verify with a benchmark that names the device before configuring anything else.
- 31Downloading models without wasting a dayDownload one quant with an include filter, pin the revision when the result matters, and check the two config files before committing to gigabytes — model repositories are mutable and most of them contain several copies of the same model.
- 32Chat templates: the most common silent failureThe model receives one formatted string, not a message list, and a wrong template or a missing stop token produces exactly the symptoms people blame on the weights — print the actual prompt before concluding anything.
- 33Context settings that truncate in silenceTrained length, allocated window and what the client sends are three different numbers, and the smallest wins silently — check the prompt token count against the configured window before blaming the model's memory.
- 34The API every engine speaksEvery local engine speaks the same HTTP dialect, so code moves between local and hosted unchanged — but compatibility is a claim about shape, and unknown parameters are usually accepted and silently ignored.
- 35Output that parses every timeA grammar makes invalid output impossible by zeroing illegal tokens at each step, but it guarantees only shape — put a reasoning field first, keep schemas small, and always include a way for the model to say it does not know.
- 36Embeddings and rerankers on your own machineEmbedding models are small enough to run on any CPU and are the foundation of local retrieval, but similarity is not relevance — pair them with a cross-encoder reranker and exact matching, and expect negation to be invisible.
- 37Speech, vision and the rest of the local stackTranscription, embedding, reranking and speech synthesis all run on an ordinary CPU, so a complete private stack needs a GPU only for the language model — and a vision model needs its projector file or it loads without being able to see.
Module 5
Serving engines and performance
Once more than one request exists at a time, the engine's scheduler matters more than the model. This block takes apart the machinery that separates a serving stack from a loop around a model: iteration-level batching, cache blocks shared as a prefix tree, drafting several tokens at once and checking them in parallel, the attention kernels underneath all of it, and how to run a load test whose numbers mean something. It ends with an ordered tuning checklist and what to do when one machine has to hold more than one model.
By the end you can
Configure and tune a serving engine for a stated load — batching, cache blocks, prefix reuse, speculative drafting — measure it with an open-loop load test reporting percentiles rather than averages, and say which single change would raise the number that is actually limiting you
- 38Serving engines, and what "faster" meansChoose an engine by your concurrency pattern; single-stream speed and aggregate throughput are different, opposing numbers.
- 39Continuous batching, and the queue underneath itContinuous batching re-forms the batch after every decoding step, so a new request waits milliseconds rather than for the previous batch to drain — and preemption in the logs means your cache memory is too small for the traffic.
- 40Prefix caching, and how to design prompts for itPrefix caching reuses the KV cache for an exact leading match, so fixed content goes first and anything variable — a timestamp, a session id — goes last, or the cache never hits at all.
- 41Speculative decoding: several tokens per weight readA draft model proposes several tokens and the full model verifies them in one pass with identical output distribution — the gain is entirely governed by acceptance rate, and it disappears on a heavily batched server where there is no idle compute left.
- 42The kernels under everything: flash and paged attentionFlashAttention removes the quadratic intermediate matrix and PagedAttention removes the worst-case cache reservation — and the kernel that reads a quantised file matters as much as the file, which is why a 4-bit model can be slower than fp16.
- 43A load test whose numbers mean somethingTest with a fixed arrival rate rather than fixed concurrency, with prompt and output lengths drawn from real traffic, and report p95 alongside the median — a closed-loop test on uniform prompts cannot show you the queue that will actually break the service.
- 44The tuning checklist, in orderTune in order — accelerator active, fast kernels, model fully resident, context sized, cache quantised, prefix caching, then batching — changing one thing at a time against a named number and writing down what happened.
- 45One machine, several modelsAdd the model sizes before designing anything — embedding and reranking models are small enough to stay resident, and several fine-tunes of one base should be served as adapters rather than as separate models.
- 46Queues, timeouts and refusing work gracefullyPast capacity, an unbounded queue turns a throughput problem into an outage — cap the queue, refuse fast with 429, cancel on client disconnect, and decide the fallback before the incident rather than during it.
Module 6
Getting good output
When a local model disappoints, the weights are the last thing to blame and usually the first thing blamed. This block works through everything between the model and the text: the samplers that choose each token and the newer ones that fix repetition without breaking syntax, how prompting a small model differs from prompting a large one, tool calls that actually parse, what long context does to quality, why the same seed gives different answers, and a diagnostic ladder that finds the real cause in the right order.
By the end you can
Diagnose a disappointing local output to its actual cause — sampler, template, prompt, context length, quantisation or the weights — in a defined order, and build a small evaluation harness that tells you whether any change you make is an improvement or noise
- 47Sampling: why output quality is often not the modelThe sampler chooses; before blaming the model, check the template, the context, and one knob at a time.
- 48The samplers most interfaces do not show youDRY penalises repeated sequences rather than repeated tokens, which is why it suppresses looping without breaking brackets and indentation the way repetition_penalty does at any effective setting.
- 49Prompting a small model is a different craftSmall models follow demonstrations better than descriptions and short ordered lists better than paragraphs — three to five instructions per call, format shown by example, and the key constraint repeated at the end where it is acted on.
- 50Examples, anchoring and getting the shape rightDemonstrate rather than describe, cover the awkward cases in the examples, and end the prompt where the answer starts — anchoring the assistant's first tokens removes formatting failures without any grammar machinery.
- 51Tool calling that survives contact with a small modelTool calling is ordinary generation plus a parser, so match the parser to the model family, constrain the syntax with a grammar, validate the meaning in code, and cap the chain — reliability compounds and no prompt fixes an exponent.
- 52Long context in practiceAdvertised context is a capacity and not a competence — measure your own working limit with a multi-fact test, and prefer five retrieved passages to a hundred pages even when everything fits, because irrelevant text dilutes attention.
- 53Why the same seed gives different answersTemperature 0 removes sampling randomness but not floating-point non-associativity, so identical requests can still diverge when two tokens are nearly tied — pin every version, log what produced each output, and assert on properties rather than exact strings.
- 54The diagnostic ladder for a bad outputWork down the ladder in order — hardware, prompt string, stop tokens, sampler at temperature 0, quantisation, prompt design, then the weights — because the cheap causes are also by far the most common.
- 55A small harness that tells you whether a change helpedThirty frozen items with structural and property checks, run at temperature 0 and logged with latency and token counts, converts every question in this course from an argument into a ten-minute run.
Module 7
Serving it to other people
Turning a model that runs on your desk into an endpoint colleagues, an application or the public can call — reached over a tunnel rather than an open port, fronted by something that authenticates and counts, supervised so it survives a reboot, logged without becoming a privacy liability, and moderated by you, because there is nobody else.
By the end you can
Expose a local model as a service other people can use safely — bound to loopback and reached over SSH or a WireGuard mesh, fronted by a gateway that holds keys and quotas, packaged and supervised so it restarts on its own, logging what is needed without retaining prompts you cannot defend — and state what an unauthenticated inference port costs you within hours of being reachable
- 56Serving it to other peopleBind to localhost and reach it over a private network; an open inference port is free compute for strangers.
- 57Reaching the machine from somewhere elseThe bind address decides who may connect and reachability decides who can, so keep the server on loopback and carry the connection over SSH or a WireGuard mesh — an unauthenticated inference port is found by scanners within hours, and the abuse that follows is attributable to your machine.
- 58A gateway in front: keys, quotas and a billInference engines implement inference and nothing else, so identity, quotas and audit belong in a gateway that holds per-person keys, enforces requests, tokens and concurrency as three separate limits, caps output length where the caller cannot raise it, and forwards only the inference paths.
- 59Containers, drivers and the version matrixThe host driver sets the ceiling and the container carries the CUDA runtime, so newer containers run on older hosts but never the reverse — verify the GPU with a bare CUDA image first, mount weights rather than baking them in, and pin the engine tag so a default cannot change underneath you.
- 60Running it as a service that survives a rebootA supervisor turns a terminal process into a service, but the details that matter are a long start timeout so a large model can finish loading, a warm-up request so the first user does not pay for disk reads, and a look at the kernel log whenever it restarts — because an out-of-memory kill leaves no trace in the engine's own logs.
- 61Logging without building a liabilitySelf-hosting moves prompt-retention risk from a vendor's policy to yours, so log operations and accounting without content, make content logging opt-in and short-lived, and verify what is actually stored by sending a canary string and grepping every path the request touched.
- 62You are now the moderation teamOpen weights ship with trained refusals and nothing else, so serving them to others means choosing deliberately among model alignment, classifier passes that cost a real forward pass each, and deterministic quotas — and treating prompt injection as unsolved, defended by architecture rather than by instructions.
- 63One card, several peopleEngines schedule first-come, first-served, so fairness on a shared card comes from per-key concurrency caps and a priority queue with ageing to prevent starvation — and routing a conversation back to the replica holding its prefix cache is free throughput that round-robin discards.
- 64Local first, API behind itRunning local first with an API behind it works when the routing rule is explicit — split by task for the high-volume cheap work, escalate on a deterministic check rather than the model's own confidence, and measure the deflection rate, because a cascade that escalates most of the time costs more than not having one.
Module 8
What it actually costs
The arithmetic that decides whether running it yourself is cheaper: duty cycle, electricity at your own tariff, the card as a depreciating asset, rented hours billed in wall-clock, and the hours nobody invoices — assembled into a break-even volume rather than an opinion.
By the end you can
Produce a defensible cost per million tokens for a stated local deployment — capital amortised over a holding period you can justify, electricity at your own tariff, duty cycle measured rather than assumed, and your own time priced — then express the comparison with a hosted API as the monthly token volume at which local begins to win, and say which input the answer is most sensitive to
- 65The honest cost comparisonLocal cost per token is set by how busy the GPU is; an idle card generates the most expensive tokens there are.
- 66Duty cycle, and why an idle card is expensiveCost per token is fixed cost divided by tokens actually produced, so duty cycle moves the answer by a factor of ten while card price and electricity move it by tens of per cent — measure it from the gateway's counters and a saturating load test, and raise it with batch work rather than invented work.
- 67Electricity, heat and the power limit knobElectricity is close to a rounding error per token when the card is busy and the dominant waste when it idles, so measure watts at the wall, cap board power for a disproportionate saving, and check the throttle reasons before blaming software for a gradual loss of speed.
- 68Buying hardware, in the right orderBuy VRAM first, bandwidth second and compute third, because a model that does not fit cannot be made to fit by being fast — and rent for a month before buying, since the workload people predict is reliably larger than the one they measure.
- 69Renting a GPU, and the meter that runs while you thinkRented GPUs bill wall-clock time rather than tokens, so the engineering is to keep weights on a persistent volume, run jobs as scripts that terminate themselves, and reserve interruptible capacity for work that restarts automatically — and renting gives up the privacy argument unless you trust whoever administers the machine.
- 70The card as an asset, and the sunk-cost trapCapital cost is purchase minus expected resale divided by the months you will actually hold the card, so the amortisation window moves the answer more than the card choice does — and a card you already own has no capital cost in the decision to use it, which is a different question from whether to buy one.
- 71The hours nobody invoicesA person's time is usually the largest first-year cost of self-hosting and the only line nobody writes down, so price the twenty to forty hours of setup and the monthly maintenance honestly — and treat a stack only one person can restart as the risk that most often sends an organisation back to an API.
- 72Building the break-even modelAssemble the inputs into two cost curves and report where they cross as a monthly token volume translated into daily exchanges, because the crossing point is usually far higher than people expect, duty cycle and unpriced hours dominate every other input, and the non-cost reasons for local belong outside the arithmetic.
- 73When the API simply wins, and when it cannotHosted wins on frontier capability, low or spiky volume, operational burden and pace of improvement, while local wins where data cannot leave, connectivity is unreliable, latency must be local, cost must be predictable or a payment card is not available — so start hosted, measure for a month, and move only the work that crosses a line you computed.
Module 9
Making it yours: adapters and training
When a prompt is not enough and retrieval is the wrong tool: what a LoRA adapter actually is, how to train one on a single consumer card, how to build the few hundred examples that decide the result, how to tell generalisation from memorisation, how to serve or merge what you made, and what licence the weights you produced actually carry.
By the end you can
Decide whether a stated failure calls for a better prompt, retrieval or weight training; produce a LoRA adapter on one consumer card from a dataset you built with the correct chat template and loss masking; distinguish generalisation from memorisation and from catastrophic forgetting using a held-out set and a general-capability check; serve or merge the adapter; and state which licences the resulting weights inherit
- 74Prompt, retrieval or training: choosing the right toolFine-tuning changes behaviour and does not install facts, so a knowledge gap belongs to retrieval and a shape problem belongs to the prompt — train only for a durable behaviour, and only after writing down the measurement that will tell you whether the run worked.
- 75What a LoRA actually isA LoRA replaces a full weight update with the product of two thin matrices, so only a fraction of a per cent of the parameters train and the frozen base needs no optimiser state — rank sets capacity, the alpha-over-rank ratio sets effective step size, and adapting the MLP projections as well as attention is what makes it learn tasks rather than only style.
- 76QLoRA: training a 7B on the card you haveLoRA removes gradients and optimiser state for the frozen base and QLoRA loads that base in 4-bit NF4, bringing a 7B run onto a 12GB card — sequence length is the memory knob that bites first, and an adapter trained against a 4-bit base must either be served on that base or re-evaluated after merging into 16-bit weights.
- 77The dataset is the projectRender your training examples with the model's own chat template and read the output before training, compute loss on assistant tokens only, include refusals and awkward shapes because a dataset of successes teaches a model to always succeed — and record provenance per example while it is still possible.
- 78Running the training runLearning rate around 1e-4 to 2e-4, two epochs and an effective batch of sixteen to thirty-two covers most LoRA runs — and printing the decoded tokens that actually contribute to the loss before launching catches the data-formatting bugs that cause most first-run failures.
- 79Generalisation, memorisation and forgettingOnly evaluation loss on held-out data carries information, and neither loss curve detects catastrophic forgetting — so keep a twenty-item general-capability set from before the run, compare outputs side by side afterwards, and raise human review after a fine-tune because errors now arrive in a register reviewers trust.
- 80Serving what you made, merged or notKeeping adapters separate serves many tasks from one base in VRAM at a small per-token cost, while merging produces an ordinary model that any engine can load — and because a merge and a subsequent quantisation each change the artefact, evaluate after every transformation and version the adapter together with its exact base commit and template.
- 81When an adapter is not enoughContinued pretraining is for distribution shifts an adapter cannot cross, needs hundreds of millions of in-domain tokens, and strips instruction following so it must be followed by a second supervised stage — and for most projects the better move is finding a base model already trained for that language or domain.
- 82What you may ship, and what nobody has settledA fine-tuned model inherits obligations from the base licence, the dataset's terms and the terms of any model used to generate data — and because copyright in training and in weights themselves is unsettled and differs by country, the durable protection is a provenance record and a short model card written at the time.