Keeping the data on your own machine

The complete answer to the privacy question

If the model runs on your computer, nothing is sent anywhere. No logs, no subprocessors, no jurisdiction question, no retention policy to read. For sensitive material this is not a mitigation, it is a solution, and it has become genuinely practical on ordinary hardware in the last two years.

It is worth being precise about what "ordinary" means, because vague claims here waste people's evenings.

What actually runs on what

Models are distributed as files of weights, and the file size is the constraint. A model's parameter count times the bytes per parameter gives roughly the memory needed. Quantisation — storing each number in 4 bits instead of 16 — cuts that by roughly four times for a modest quality loss, and it is the reason any of this is possible on a laptop.

A rough guide, for 4-bit quantised models:

• A 1 to 3 billion parameter model needs about 1 to 2 GB. Runs on a phone or any laptop from the last decade. Good for summarising, extraction, classification, simple rewriting. Not good at reasoning or long documents.
• A 7 to 9 billion parameter model needs about 4 to 6 GB. Runs comfortably on a machine with 8 GB of RAM, and well on 16 GB. This is the useful floor for general work: drafting, translation, answering questions about a document you supply.
• A 13 to 14 billion parameter model needs about 8 to 10 GB. Wants 16 GB of RAM.
• A 30 billion parameter model and above needs a machine with a good discrete GPU or an Apple Silicon Mac with plenty of unified memory.

Apple Silicon Macs are unusually good at this because the processor and graphics share memory, so a Mac with 16 GB can hold models that would need a dedicated graphics card on a PC. On Windows and Linux, VRAM on the graphics card is the number that matters; system RAM works but is several times slower.

Speed on a mid-range laptop without a GPU is roughly five to fifteen words per second for a 7B model. Slower than a commercial service, fast enough to read along with.

The free software

All of the following are free, and none require a subscription or an account.

• Ollama — the simplest starting point. Install, then ollama run llama3.2 and you are talking to a model. Command line, with a local API on port 11434 that other tools can use.
• LM Studio — a graphical application for people who would rather not use a terminal. Browse models, download, chat.
• llama.cpp — the C++ engine most of the others are built on. Worth knowing the name; it is why this works on a phone at all.
• GPT4All and Jan — desktop applications with document-chat built in.
• Open WebUI — a browser interface resembling the commercial products, pointed at your local Ollama.

For models, look for the instruction-tuned variants of Llama, Qwen, Gemma, Mistral and Phi. Check the licence: several are permissive, some carry conditions such as a user-count threshold or use restrictions.

What you give up

Be honest about the trade, because people who expect parity are disappointed and abandon it.

A 7B model is noticeably weaker than a frontier commercial model at complex reasoning, long-context work, code and unusual languages. It hallucinates more. It has an older knowledge cutoff. It will not browse the web unless you add that yourself. Long documents may exceed its context window.

What it is genuinely good at is the large category of ordinary tasks: summarising a document you supply, extracting fields, rewriting, classifying, drafting a first version, answering questions about text in front of it. Which happens to be most of what people do with these tools at work.

When it is the right call

Medical, legal, HR and financial documents about identifiable people. Anything under an NDA. Journalistic material with a source to protect. Personal writing — a diary, a letter, a health worry — where the discomfort is not about legality at all. Work in an organisation with no approved cloud tool, where the real alternative is not "a compliant service" but "someone pastes it into a consumer chatbot anyway".

And one situation people underrate: a laptop on a plane, on a train, or in a village with intermittent connectivity. A local model works with the network off, which is a practical advantage before it is a privacy one.