The context window, and why a long chat gets worse

Everything it knows, every time

When you send your fortieth message in a conversation, the model does not remember the previous thirty-nine. It is handed all of them again, as one long block of text, and reads the lot from scratch before writing a word.

That block is the context window, and it holds everything: the product's hidden instructions, any documents you attached, every message you have sent, every reply it has given, and your latest question. The window has a fixed size, counted in tokens. Nothing outside it exists.

Almost every confusing behaviour in a long chat comes from this one fact.

The size, in things you recognise

Common limits today run from about 128,000 tokens to a million, with 200,000 fairly typical for a paid business tier. Converting to work you would recognise:

• A page of ordinary prose: roughly 500 tokens.
• A 40-page report: roughly 20,000.
• A 300-page contract bundle: roughly 150,000 — already over a 128,000 limit on its own.
• An hour of transcribed meeting: 8,000 to 12,000.

So a 128,000-token window will hold a substantial document and a long discussion of it, and will not hold your document library. A million-token window will hold a great deal, and costs proportionally more to use.

What happens when it fills

Products handle overflow in one of three ways, and which one yours uses changes what you should do.

Some refuse, with an error saying the conversation is too long. This is the honest failure and the easiest to work with.

Some truncate silently, dropping the oldest messages to make room. This produces the experience everybody has had: you gave a careful instruction at the start — "always keep the client's name out of this" — and twenty messages later it stops obeying. It did not forget. The instruction is no longer in front of it, and there is nothing left that knows it ever existed.

Some summarise the older part of the conversation and keep the summary. Better than dropping, and still lossy in exactly the way summaries are lossy: the specific number you mentioned in passing is the first thing to go.

The middle is the weak part

Even inside the window, attention is not even. Across models, material at the very beginning and the very end of a long context is retrieved far more reliably than material in the middle. A fact planted 60 per cent of the way through a long document is measurably more likely to be missed than the same fact at either end.

Two practical consequences. Put the instruction that matters most at the end of your message, immediately before you ask for the output, not buried at the top of a long brief. And when you are working from a long document, ask located questions — "in section 7, what is the notice period" — rather than one question of the whole file. The full treatment of this is later in the course; the mechanism is here.

The cost curve nobody warns you about

You are billed for every token going in, on every turn. Because the whole transcript is resent each time, the input cost of a conversation grows with the square of its length.

Work it through. A chat where each turn adds 500 tokens costs 500 input tokens on turn one, 1,000 on turn two, and 20,000 on turn forty. The forty-turn conversation has consumed around 400,000 input tokens in total, not 20,000. Attach a 50-page document at the start and every single turn afterwards re-reads all 25,000 tokens of it.

At a representative business rate of a few dollars per million input tokens, a single long working session over a large document is cents rather than dollars — but a team of thirty doing it daily is a real line on a bill, and it is why "why is our usage so high, nobody is doing anything unusual" has a boring answer. Prompt caching, offered by most providers, cuts the cost of the repeated part substantially and is worth asking your administrator about.

Four habits that follow

1. One conversation per task. When you move to a different job, start a new chat. You are not being wasteful; you are removing thousands of tokens of irrelevant history that are both costing money and competing for attention.
2. Restate the constraint late. If it has drifted, do not scold it. Repeat the requirement in your next message, where it will be at the end of the window and most visible.
3. Attach once. Pasting the same document three times in one conversation triples its cost and gives the model three copies to confuse.
4. When it goes strange, start again. A conversation that has begun contradicting itself is usually one where the early material has been dropped or summarised away. Ten minutes of arguing with it will not recover the missing text. A fresh chat with a clean brief will.

The free path is the same one: run a local model with a small window and you can watch the truncation happen in a few minutes rather than inferring it from a bad afternoon.