What the machine is actually doing

There is no encyclopedia inside

The most common wrong picture of a language model is a very fast librarian: you ask a question, it looks the answer up, it reads it back. Almost every confusion in this course dissolves once you replace that picture with the real one.

A model is a large pile of numbers — its weights. For a model you can download today that might be seven billion numbers; for the largest commercial ones, hundreds of billions. Those numbers were adjusted, over weeks of computation, so that when text is fed in, the arithmetic that comes out predicts the next chunk of text well. Nothing else was optimised. Not truth, not helpfulness, not caution. Just: given what came before, what usually comes next.

When you type a question, the text is chopped into tokens — roughly, common word fragments. "Unbelievable" might be three tokens. Those tokens run through the arithmetic and the model produces a probability for every possible next token: perhaps 0.31 for one, 0.12 for another, and small numbers spread across fifty thousand more. One is picked. It is added to the text, and the whole thing runs again for the token after that. A page of output is that loop, several hundred times.

Why this explains so much

Hold that mechanism next to the failures and they stop being mysterious.

It has no separate store of facts to check against. Facts, grammar, style and nonsense are all encoded in the same weights, in the same way. There is no line in the code where truth is looked up, so there is nowhere to put a check that says "only output this if true".

It cannot tell you it does not know. Every prompt produces a probability distribution. There is no state called empty. The distribution for a question about a real 2019 paper and the distribution for a question about a paper that never existed are both perfectly ordinary distributions, and both produce fluent text.

Fluency is cheap and facts are expensive. Grammar is enormously regular — it appears in every sentence of the training data, so the statistics for it are excellent. A specific fact may have appeared four times in a trillion words. The model is superb at the first and improvising at the second, and the output looks identical either way.

Its behaviour is not a rule you can read. Nobody wrote "be polite" or "decline that request" as a line of code you could open and inspect. Those behaviours were trained in, and trained behaviours can be worked around in ways that written rules cannot.

Training, fine-tuning, and the thing you talk to

Three stages, and mixing them up causes real confusion.

Pre-training is the expensive one. Enormous quantities of text, months of computation, and the result is a model that continues text plausibly. A raw pre-trained model is not an assistant. Ask it a question and it might produce five more questions, because that is what a list of questions usually looks like.

Fine-tuning teaches the shape of an assistant: instructions get followed, questions get answered, some requests get declined. This is a comparatively small amount of curated data on top of the enormous base.

Alignment training — usually some form of learning from human preference ratings — pushes the model towards responses people rated well. Notice what that optimises: rated well, by people reading quickly. Answers that sound confident and agreeable rate well. A large part of the sycophancy you will meet later in this course arrives here, honestly earned by the training objective.

Then there is the layer you never see. The product you use is the model plus a hidden system prompt, plus filters on input and output, plus sometimes a search step. When behaviour changes overnight and the model has not been updated, this layer is usually why.

The one test to remember

When something surprises you, ask: would this behaviour be produced by a system that predicts likely text and has no concept of truth? Confident fabrication: yes. Agreeing with you when you push back: yes. Getting a well-known fact right and an obscure one wrong: yes. Deliberately deceiving you for its own ends: that requires a much larger claim, and the ordinary explanation covers what you are seeing.

That test will not answer every question. But it will keep you from the two failure modes of AI commentary — treating the system as a person with motives, and treating it as a database with a bug.