What "chat with your documents" is really doing

"Chat with your documents" does not mean it read your documents

When you point a tool at a folder of 400 files and ask a question, the model does not read 400 files. It cannot — they do not fit in the context window, and reading them all for every question would be absurdly expensive.

What happens instead is a search step, and understanding it explains every strange result you have ever had from one of these tools.

The five steps

1. Split. Your documents are cut into chunks, typically a few hundred words each, sometimes with a little overlap.
2. Embed. Each chunk is converted into an embedding — a list of several hundred or a few thousand numbers that places the chunk in a space where passages with similar meaning sit close together. "Termination of employment" and "ending a contract of service" land near each other despite sharing no words. That is the whole trick, and it is genuinely useful.
3. Embed your question the same way.
4. Retrieve. The system finds the nearest handful of chunks — often three to ten. This number is called top-k.
5. Answer. Those chunks, and only those chunks, are pasted into a prompt with your question.

So the model's view of your 400 files is five paragraphs. Everything it says is either in those five paragraphs, or it is coming from general training knowledge, and the reply does not distinguish the two.

Why the right passage gets missed

Exact identifiers embed badly. An invoice number, a part code, a case reference or an unusual surname carries almost no semantic content, so it does not sit near anything in meaning-space. Searching for "INV-2291" through an embedding index can fail completely where a plain keyword search finds it instantly. Good systems combine both — this is called hybrid search — and many do not. If you are hunting for a specific string, use the search box in your file manager or grep, not the chat.

A rule and its exception are different chunks. The definition on page 3 and the carve-out on page 40 are separated at split time. Retrieve one and you get half a rule, stated completely.

Counting is structurally impossible. "How many of these contracts contain an indemnity clause?" cannot be answered by retrieving five chunks. The system will retrieve five, and the model will answer as though five were all there were. You will get a number. The number is meaningless, and nothing in the answer says so.

This is the single most dangerous failure of document chat in office use, because the question sounds routine and the answer looks like a fact. Aggregate questions — how many, list all, which ones do not — are not retrieval questions. For those you need a spreadsheet, a script, or one pass over every document with the results collected.

Absence is not retrievable. "Which suppliers have no insurance clause" asks for chunks that do not exist. Nothing can be retrieved to prove a negative, and the answer you get is an inference from whatever happened to come back.

How to use it well

• Ask located questions. "In the termination section, what notice does the supplier owe us?" outperforms "what are the termination terms?" because it aims the retrieval.
• Use the document's own vocabulary. Embeddings bridge some of the gap between your words and the document's, not all of it. If the contract says "the Services", say "the Services".
• Always demand the source. "Quote the sentence and give the file name and page." Then open it. A quoted sentence you can find is a fact; a summary is a hypothesis.
• Read the sources panel. Many tools show which chunks were retrieved. If the retrieved passages are not about your question, the answer is worthless no matter how well written, and you have learned that in two seconds.
• Ask the same question three or four different ways. Different phrasings retrieve different chunks. When they produce different answers, the retrieval is unstable, not the document.

The alternative that avoids all of this

If your document fits in the context window, do not use retrieval at all. Paste the whole thing.

A 50-page document is around 25,000 tokens and fits comfortably in a 128,000-token window. There is then no search step, no chunk boundary and no missed passage — the model sees everything. It costs more per question, because you are paying for the whole document each time, and it is dramatically more reliable.

The rough rule: under about 50 pages, paste it whole. Above that, retrieval, with the discipline above. And for a question that spans a whole library, decompose it — one pass per document, results collected by you.

Building your own, free

You do not need to. But if the material may not leave your building, the whole stack is available free: Ollama to run both a chat model and an embedding model such as nomic-embed-text locally, and a small amount of code — or one of the free local document-chat applications built on this — to do the splitting and searching. It runs on a laptop and nothing goes anywhere.

And keep grep in mind. For "does this exact phrase appear anywhere in these 400 files", a forty-year-old text search tool beats every embedding system ever built, in about a second.