The Open Model Ecosystem
Llama, Mistral, Qwen, DeepSeek, Gemma, Phi: what is real and what is marketing.
- Level
- Some background helps
- Lessons
- 71
- Reading time
- 625 min
- Price
- Free, no sign-up to read
Llama, Mistral, Qwen, DeepSeek, Gemma and Phi are not interchangeable, and the leaderboard that ranks them is mostly measuring itself. This course teaches what open weights actually gets you, how to read the licence and the model card, how to size a model to the machine you really have, and how to settle a model choice with forty of your own examples in one afternoon.
Opens after the The Maths You Actually Need exam
Sign in, finish that course, and pass its exam. You can read this syllabus meanwhile.
Go to The Maths You Actually NeedModule 1
What open actually means
The word open is doing a great deal of work in this ecosystem, and it means something different in every sentence it appears in. This block separates the freedoms a release actually grants — run it, adapt it, redistribute it, inspect it, rebuild it — from the ones the announcement implies, and gives you the licence chain, the data question, the regulatory position and the supply-chain checks that a model you are about to depend on has to survive.
By the end you can
Given any model release, say precisely which freedoms it grants and which it withholds, trace its licence through every fine-tune in the chain, verify the files you downloaded are the files the publisher made, and state what an auditor or a regulator could still ask that the weights alone cannot answer
- 1Open Weights Is Not Open SourceOpen weights means you can run and adapt the model, not that you can see how it was made.
- 2Reading the Licence Before You ShipA licence travels with every fine-tune and every download; check the chain, not the badge on the page.
- 3Why a lab gives away a modelA lab releases weights when a cheap, ubiquitous model helps something else it sells, so openness tracks strategy rather than principle and can be withdrawn from the next version at any time.
- 4Grading openness without arguing about the wordScore a release on six separate axes — weights, licence, data, code, evaluation, checkpoints — because each axis buys a different concrete capability, and only some of them matter for any given job.
- 5Where the training data comes fromOpen models are built on a handful of named public web corpora processed by undisclosed filters, and crawler opt-outs only ever work forwards, so what a model already learned cannot be withdrawn.
- 6The copyright question nobody has settledCopyright and AI splits into three independent questions — whether training infringes, whether an output infringes, and whether an output can be owned — and they are answered differently in every jurisdiction and mostly still unresolved.
- 7What regulators ask of an open modelThe EU's open-source exemption for general-purpose models still requires a copyright policy and a public training-data summary, does not apply above the systemic-risk compute threshold, and publishing a fine-tune can make you a provider with duties of your own.
- 8Trusting a file you did not buildSafetensors removes only one of the execution paths into your machine; trust_remote_code runs repository Python by design, and behavioural backdoors in weights or adapters are invisible to every scanner.
- 9Vetting a stranger's fine-tuneVet a community fine-tune on its licence chain, its named datasets, its stated limitations and its performance on your own examples, because download counts and benchmark tables without a harness carry almost no information.
- 10Publishing weights people can trustA trustworthy release is the base model's score reported next to yours under an identical, named configuration, plus a working chat template, an honest limitations section and a licence chain you actually have the right to grant.
Module 2
The map, and how to read it
Open weights are not only chat models. There are code models with their own token formats, embedding models that decide whether search works, rerankers, speech models, image models with the ecosystem's worst licence traps, regional models that beat much larger general ones in their own languages, and a market of providers serving all of it by the token. This block draws the map, names what is durable about each part, and tells you what a model of a given size can honestly be asked to do.
By the end you can
Place any open release on a map of families, modalities, formats and serving options within ten minutes — naming its licence posture, its likely tokenizer, the tools that will serve it and the size class it competes in — and pick the right kind of model for a retrieval, code, speech or generation task rather than reaching for a chat model by reflex
- 11The Families, and What Each Is Actually ForFamilies differ durably in licence, size ladder and ecosystem; whichever leads on quality changes every few months.
- 12Hugging Face as InfrastructureNever load a pickle you did not create; prefer safetensors, and pin the commit rather than the branch.
- 13Regional and specialist modelsA language-focused model wins by knowing more of that language and spending fewer tokens on it, so compare tokenizer output as well as quality — and check the licence, because several of the best multilingual releases are non-commercial.
- 14Models that write codeCompletion models trained with fill-in-the-middle and chat-tuned instruct models are different tools with different prompt formats, and code benchmarks like HumanEval are saturated enough that twenty diffs from your own repository are better evidence.
- 15Embeddings you can host yourselfAn embedding model is small, cheap and CPU-friendly, but its query and passage prefixes, normalisation, truncation limit and pooling are set by the model card, and getting any of them wrong degrades retrieval silently rather than raising an error.
- 16Rerankers, and why retrieval is two stagesA bi-encoder must compress each document before seeing the query, which is why retrieval is fast and imprecise; a cross-encoder reranker reads query and document together, which is why it is accurate and can only reorder what retrieval already found.
- 17Speech, in both directionsWhisper invents text during silence unless voice activity detection gates it, so a production transcription pipeline is VAD plus a fast reimplementation, and text-to-speech is where the open licences get genuinely restrictive.
- 18Image and video weights, and their licence trapsOne image-model family often ships variants under completely different licences, and the widely used FLUX.1 [dev] is non-commercial while [schnell] is Apache 2.0, so check the variant rather than the family before anything reaches a customer.
- 19What a model of each size can honestly doStored knowledge scales with parameters and multi-step reliability compounds, so a small model fails on facts and long chains rather than on single narrow tasks — and narrowing the task usually beats increasing the size.
- 20Renting open weights by the tokenProviders serving the same open model differ in quantisation, context cap and default sampling, so a benchmark run against one provider does not transfer to another unless you pin the provider and record its serving configuration.
Module 3
Inside the files
A model repository is a small number of files that between them fix the memory the model needs, the cost of every request in your language, the exact conversation format it will accept and the point at which it stops talking. This block reads those files properly: the config, the tokenizer, the chat template, the attention layout, the context claim, and the formats you convert between — so that you can predict a model's behaviour before you load it, and diagnose it from the files when it misbehaves.
By the end you can
Open a model repository you have never seen and predict, from config.json and the tokenizer alone, its parameter count, its memory footprint at a given context length, its per-token cost in your languages and the exact conversation format it requires — then verify each prediction with a command and diagnose a runaway or garbled generation from the files rather than by guessing
- 21Base, Instruct, ReasoningReasoning models buy accuracy on multi-step problems with tokens and latency; most production tasks are not multi-step.
- 22Every number in config.jsonconfig.json contains everything needed to derive a model's parameter count, its weight memory and its KV cache per token, and most of the parameters live in the feed-forward matrices rather than in attention.
- 23Why the same sentence costs different amountsFertility — tokens per word — varies by a factor of three or more across languages and tokenizer vocabularies, and it multiplies straight into your bill, your effective context length and your latency.
- 24The chat template is a contractAn instruct model was trained on an exact token sequence that ships with it in tokenizer_config.json, so build prompts with apply_chat_template and add_generation_prompt rather than by hand, and watch for a duplicated beginning-of-sequence token.
- 25Attention variants, and the cache that eats your memoryKV cache memory is two times layers times key-value heads times head dimension per token, it grows with every concurrent user, and grouped-query attention, cache quantisation and a lower context limit are the three levers that change it.
- 26What a 128k context window really meansAdvertised context comes from extended positional encodings, but the effective length on tasks harder than finding one sentence is often a fraction of it, and prefill cost grows with the square of prompt length.
- 27Mixture of experts, and the memory it still needsA mixture-of-experts model computes like its active parameter count and occupies memory like its total, so it is outstanding on a machine with spare memory and useless on one without.
- 28Models that can seeA vision-language model turns image patches into tokens through a projector, so images consume context and cost proportional to resolution, and the reliable failures are counting, dense layout and small text rather than description.
- 29Formats, and what conversion costs youConversion between weight formats is where chat templates, stop tokens and unrecognised architectures get silently mangled, so prefer official conversions and verify a converted file against the original on the same prompts.
- 30Why it will not stop talkingRunaway generation is almost always a stop condition rather than a bad model — several end-of-turn tokens, a template that never opened the assistant turn, a lossy conversion, or a serving layer ignoring generation_config.json.
Module 4
Making it run on the machine you have
Most writing about local models assumes a graphics card that most readers do not own. This block does the arithmetic instead: what quantisation actually does to the numbers, which bit width to pick, how to predict tokens per second from memory bandwidth before you download anything, how to serve many users on one GPU, where to get compute for nothing, when renting stops being cheaper than owning, and what genuinely runs on a phone.
By the end you can
Size, quantise and serve an open model on hardware you actually have — a laptop, a phone, a free cloud tier or a rented GPU — predict its generation speed from memory bandwidth before running it, choose a bit width you can defend, and state the monthly token volume at which renting stops being cheaper than owning
- 31Choosing a Model for the JobFit the constraints first, memory and latency and licence and language, then test the two or three models left standing.
- 32What quantisation actually doesQuantisation stores weights as small integers with a shared scale per block, so a 4-bit model is really about 5 bits per weight, and because larger models carry more redundancy, a bigger model quantised harder usually beats a smaller one at full precision for the same memory.
- 33Choosing a bit width you can defendSpend memory on parameters before precision down to about four bits, but expect quantisation to damage long-context retrieval and non-English performance well before it damages short English chat, so test at the bit width you will ship.
- 34llama.cpp, end to endllama.cpp is the engine under most desktop AI tools, and its important dials are the number of layers offloaded to the GPU, the context length that sets your cache allocation, and llama-bench reporting prefill and decode speed as two separate numbers.
- 35Ollama and the friendly pathWrappers like Ollama choose your quantisation, sampling defaults and context length for you, and the historic 2048-token context default is the single most common cause of a local model appearing to ignore the start of a document.
- 36Predicting speed before you downloadDecoding reads every active weight from memory once per token, so tokens per second is bounded by memory bandwidth divided by model file size, while prefill is compute-bound and therefore one or two orders of magnitude faster.
- 37Serving many users at onceContinuous batching and paged attention let one GPU serve many sequences because weights are read once per step and cache memory is allocated in blocks rather than reserved per request, trading a little per-user latency for several times the total throughput.
- 38Running all of this without moneyKaggle's weekly GPU hours, Colab's T4, free CPU instances and provider credits are enough to evaluate, fine-tune and batch-process without spending anything, provided every job checkpoints to durable storage and writes results incrementally.
- 39Rent, or own: the arithmeticBreakeven against a rented GPU sits in the hundreds of millions of tokens a month and assumes high utilisation, so the real reasons to self-host are usually privacy, offline operation and permanence rather than cost.
- 40What actually runs on a phoneMobile operating systems cap per-application memory, which limits on-device models to roughly 1 to 3 GB, and sustained generation is halved by thermal throttling — so a phone model is a private text-transformation engine, not a knowledgeable assistant.
Module 5
Getting good output
An open model arrives with none of the scaffolding a commercial product puts around it: no hidden system prompt, no server-side reformatting, no safety layer, no tool harness. That is the freedom and it is also the work. This block covers the sampler, constrained decoding, prompting that survives a small model, examples, tool calls, agent loops, other languages, refusals, guardrails, and what to do when the model does not know.
By the end you can
Take an open model producing bad output and identify whether the fault lies in the sampler, the prompt, the examples, the schema, the language or the model itself — then apply the cheapest correct fix, from a temperature change to a grammar-constrained decoder to a separate guardrail model, and demonstrate the improvement on a held set
- 41The sampler, and the defaults that ship brokenThe sampler converts logits into a token, and the common default of a repetition penalty above 1 actively damages JSON and code because those formats require repeated punctuation tokens.
- 42Making the output valid by constructionGrammar-constrained decoding masks illegal tokens at each step so invalid syntax becomes impossible, which converts parse failures into silently wrong values unless the schema gives the model an explicit way to express absence.
- 43What changes when nothing is hiddenWith open weights you supply the scaffolding a hosted product hides, so instructions belong at the end of the prompt, phrased positively, one job per call — and prompts must be versioned per model because they do not transfer between families.
- 44Examples, and how to choose themExamples mainly teach the output format and the label space rather than the mapping, so consistency of format and balanced coverage of labels matter more than how subtle the individual answers are.
- 45Tool calling, and what breaks at 8BTool calling is a trained output format plus a matching server-side parser, and small models fail specifically on long tool lists, nested arguments and deciding not to call — so validate arguments and return errors as tool results rather than exceptions.
- 46Agent loops on models that are not cleverAgent loops on small models fail by never stopping, repeating actions, inventing observations, losing the goal and outgrowing the context — so hold the state, the deduplication and the step cap in code and reduce the number of model decisions rather than enlarging the model.
- 47What these models really do in your languageLanguage support divides into well-supported, thin and nominal tiers with a steep cliff between the last two, script and register matter as much as language, and only a native speaker rating thirty real inputs will tell you which tier you are in.
- 48Refusals, and the uncensored questionRefusal is generated behaviour mediated by a direction in activation space that can be removed from open weights in an afternoon, so a boundary your product needs must be enforced outside the model rather than by its tuning.
- 49Guardrails as a separate layerA separate classifier is a different failure surface from the generating model because it labels rather than obeys, so layer cheap deterministic checks, a small input classifier, the model and an output check — and never let any of it stand between a model and an irreversible action.
- 50When the model does not knowA model always produces its most likely continuation, so abstention must be made available in the schema and rewarded in the prompt, and requiring a verifiable quotation turns invisible fabrication into a check that code can fail.
Module 6
Adapting a model to your work
Fine-tuning is the freedom that open weights actually deliver, and it is also the most over-reached-for tool in the field. This block gives the decision rule for when it is warranted, the arithmetic of LoRA and QLoRA, how to build a dataset that does not encode your own inconsistency, a run you can complete on free hardware, preference tuning, distillation, model merging, the forgetting you have to measure, and how to ship the result.
By the end you can
Decide with evidence whether a task needs fine-tuning at all; if it does, build the dataset, size a LoRA to fit free hardware, run it, and demonstrate on a held-out set and a regression suite that the adapted model is better at the task and no worse at everything else
- 51When not to fine-tuneFine-tuning reliably changes form and unreliably changes knowledge, so the honest test is whether it beats your best prompt plus retrieval by a margin that justifies re-tuning every time the base model is superseded.
- 52What a LoRA actually isLoRA freezes the base and trains a low-rank product added to each targeted matrix, so under 1% of parameters carry gradients — and rank, alpha and which modules are targeted are the three settings that decide whether the run does anything.
- 53Fitting a fine-tune in 16 GBTraining memory is weights plus gradients plus optimiser state plus activations, and QLoRA collapses the first three by freezing a 4-bit base and training a small adapter — leaving activations, controlled by sequence length, batch size and gradient checkpointing, as the dial you actually turn.
- 54Five hundred examples, done properlyA fine-tune inherits every property of its dataset, so diversity of real inputs, byte-identical output format, measured label agreement and deduplicated splits matter far more than the number of examples.
- 55A real run, on free hardwareA LoRA run is thirty lines and an hour on free hardware, but the loss curve is only diagnostics — the result is the task metric on a held-out set compared against the base model with your best prompt, at the same settings.
- 56Teaching it which answer is betterPreference tuning learns from pairs or binary judgements about which output is better, fixing verbosity, tone and refusal calibration that supervised tuning cannot — at an order of magnitude lower learning rate and with a strong tendency to make models longer.
- 57Distilling a larger model into a smaller oneResponse distillation is supervised fine-tuning on a teacher's outputs, so the student inherits the teacher's errors and its style, cannot exceed it, and must be measured against ground truth rather than against agreement with the teacher.
- 58Measuring what you brokeFine-tuning moves the weights that held everything else, so report the task gain alongside a fifty-item regression set covering general instructions, other languages, alternative formats and refusals — and mixing general data into training usually keeps most of the gain with a fraction of the loss.
- 59Merging models, and what it really doesMerging averages fine-tuning deltas from a shared base, which works because those deltas stay in a connected low-loss region — and it is most reliable when merging your own runs and least trustworthy when a merge was selected on the benchmark it is reported against.
- 60When the model does not have your language at allContinued pretraining on raw text is the only fix for a language or notation the model never learned, it needs billions of tokens and vocabulary extension with mean-initialised embeddings, and it destroys instruction following unless followed by a fresh supervised tuning stage.
- 61Shipping the adapterAn unmerged adapter lets one base model serve many tuned behaviours and lets untuned traffic bypass the tune entirely, and a fine-tuned deployment must pin the base revision, adapter, prompt, sampler and both evaluation numbers as a single versioned unit.
Module 7
Judging models and claims
Every part of this ecosystem asks you to believe a number: a benchmark table, a leaderboard position, a quantisation that is claimed to be lossless, an embedding model at the top of a chart, a price per million tokens. This block is about generating your own evidence instead — reading a model card for what it does not say, running the standard harness yourself, proving what a quantisation cost, measuring latency and cost per task properly, and swapping a model under a live product without breaking it.
By the end you can
Settle a model choice with evidence you produced yourself — a private held-out set, a pinned harness, a quantisation compared against its full-precision reference, latency measured at real concurrency and cost expressed per task rather than per token — and explain to a sceptical colleague which published claims about that model you believe and why
- 62What Benchmarks Actually MeasureA benchmark result is evidence about that benchmark; transfer to your task is a hypothesis you have to test.
- 63Reading a model card for what it does not sayA model card's front matter is structured and checkable while its prose is a set of claims, and a benchmark number without the harness commit, shot count, aggregation, decoding settings and self-run baselines is not reproducible evidence.
- 64The leaderboards that still carry informationLeaderboards die from saturation and contamination, so the ones that still carry information use private or rotating items, execution-based verification or fresh human prompts — and even then their proper use is shortlisting, not deciding.
- 65Running the standard harness yourselfRunning lm-evaluation-harness yourself against any OpenAI-compatible endpoint is how you calibrate published claims and catch regressions, and your number will differ from a paper's mostly through shot count, chat template and harness version — so pin the commit and log the samples.
- 66Proving what a quantisation costPerplexity is a screening test; the sharper measurement is divergence from the full-precision model on the same corpus, reported as the fraction of tokens where the top choice differs, plus agreement on your own examples at long context and in your languages.
- 67Evaluating an embedding model on your corpusRetrieval is cheap to evaluate on your own corpus with 50 labelled queries, and the grid of embedding model against chunk size usually shows chunking moving recall more than the model does.
- 68Measuring latency properlyReport time to first token and per-token latency separately at p95 and at real concurrency after discarding warm-up requests, because prefill and decode are limited by different resources and the mean hides the tail.
- 69Cost per task, not cost per tokenCost is tokens per task times price times retries, so a cheaper model carrying a long few-shot prompt or a reasoning model spending output tokens on thinking can cost several times more than a dearer model with a short prompt.
- 70Swapping the model under a live productA model swap is a release, not a configuration change: compare each model with a prompt tuned for it, run the regression, refusal, cost and latency checks, shadow real traffic before canarying, and pin revisions and providers so nobody swaps the model for you.
- 71Evaluate a Model on Your Task in an AfternoonForty real examples and a fixed harness settle a model choice better than any leaderboard, in about four hours.