A free model on your own machine, called from Python

Why a local model

Three reasons, and a learner on a phone or an old laptop will feel all of them. It costs nothing per call, so you can run your script two hundred times while learning. It needs no account, no card and no key, so nothing can leak. And it works offline. What you give up is quality — a 1-billion-parameter model on a CPU is not a hosted frontier model — and speed. For learning, testing and many real tasks, that trade is good.

The arithmetic first

A model's memory is its parameters multiplied by the bytes per weight:

• 0.5B parameters × 2 bytes (float16) ≈ 1 GB
• 3B × 2 ≈ 6 GB; 3B × 0.5 (4-bit quantised) ≈ 1.5–2 GB
• 7B × 2 ≈ 14 GB; 7B × 0.5 ≈ 4 GB
• 70B × 0.5 ≈ 35–40 GB

Add a gigabyte or so of working memory. A laptop with 8 GB runs a 3B model quantised comfortably, a 7B model quantised at a squeeze, and a 7B model in float16 not at all — the process is killed by the operating system with no Python traceback, because the failure happens below Python. Do the multiplication before the download.

Speed on a CPU is a few to a few dozen tokens per second depending on size and machine. A 0.5B model answers in a couple of seconds; a 7B model quantised writes at reading speed on a recent laptop and slower on an old one. A phone under Termux runs 0.5B to 1.5B models.

Ollama: the easy path

Ollama is a free program that downloads models, quantises them and serves them over HTTP on your machine.

ollama pull qwen2.5:1.5b        # ~1 GB download
ollama run qwen2.5:1.5b         # chat in the terminal to check it works

From Python, you already know how to talk to it — module 6 did, through the OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(model="qwen2.5:1.5b",
        messages=[{"role": "user", "content": "Explain a dict in one sentence."}])
print(r.choices[0].message.content)

Or with plain requests against Ollama's own API, which also streams:

r = requests.post("http://localhost:11434/api/chat", json={
    "model": "qwen2.5:1.5b",
    "messages": [{"role": "user", "content": "Hi"}],
    "stream": False,
}, timeout=120)
print(r.json()["message"]["content"])

ollama list shows what you have; models are stored under ~/.ollama. Embedding models — nomic-embed-text, bge-m3 for multilingual — run the same way, via /api/embeddings, so the notes search from the previous lesson can be entirely local.

llama.cpp: the engine underneath

Ollama wraps llama.cpp, a C++ inference engine that runs quantised models in the GGUF format on almost anything. Its Python binding gives you the model as an object with no server:

from llama_cpp import Llama

llm = Llama(model_path="qwen2.5-1.5b-instruct-q4_k_m.gguf", n_ctx=4096, verbose=False)
out = llm.create_chat_completion(messages=[{"role": "user", "content": "Hi"}], max_tokens=200)
print(out["choices"][0]["message"]["content"])

GGUF files come from Hugging Face; the q4_k_m in the name is the quantisation — 4-bit, medium, the usual choice. n_ctx is the context window you allocate, and memory grows with it. Use this when you want a single self-contained script with no background process, or on a machine where you cannot install a service.

transformers: the research path

Hugging Face's transformers loads the original weights, unquantised, through PyTorch:

from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")
out = pipe([{"role": "user", "content": "Hi"}], max_new_tokens=100)
print(out[0]["generated_text"][-1]["content"])

This is slowest on a CPU and uses the most memory, and it is what you need when you want to inspect the model — its tokeniser, its attention, its logits — or fine-tune it, which is its own course. For plain generation, prefer Ollama or llama.cpp.

The provider class, again

Module 7's ChatProvider is why none of this disturbs the rest of a program:

class OllamaProvider(ChatProvider):
    def __init__(self, model="qwen2.5:1.5b"):
        self.client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
        self.model = model
    def complete(self, messages, max_tokens=500):
        r = self.client.chat.completions.create(model=self.model, messages=messages, max_tokens=max_tokens)
        return r.choices[0].message.content

Develop against this, run the tests against this, and switch to the hosted provider for the final run or when quality matters. A budget guard from module 6 that falls back to the local model when the cap is near is composition doing exactly what it was for.

What to expect from small models

They follow simple instructions well and complicated ones badly. They produce valid JSON less reliably, so module 6's validation and retry matter more. They know less, and they invent more confidently. They handle English best and Hindi variably; the Qwen and Gemma families are stronger on Indian languages than most at the same size. None of this is a reason not to use them; it is a reason to keep the validation layer and to test the prompt on the model you will actually run.

Try this now

Do the memory arithmetic for your machine, pull the largest model that fits with room to spare, and time a 200-token reply. Then write OllamaProvider, run your chat loop against it, and swap in the fake provider from module 7 to confirm nothing else changed.