Files larger than memory

The line that kills the process

data = open("events.jsonl").read()          # a 6 GB string
rows = list(csv.DictReader(f))              # 40 million dictionaries

Both load the whole file into memory. On a laptop with 8 GB of RAM the first raises MemoryError if you are lucky, and if you are not, the operating system starts swapping and the machine becomes unusable for ten minutes.

A Python object costs far more than the bytes it represents. A dictionary with five short string keys is roughly 300–400 bytes; the same row in the file might be 60. Loading a 1 GB CSV into a list of dictionaries can need 6–8 GB.

Files are already iterators

with open("events.jsonl", encoding="utf-8") as f:
    for line in f:
        record = json.loads(line)
        process(record)

This holds one line at a time. The file object reads a buffer from disk, hands you lines, and discards them as you go. It works identically for a 2 KB file and a 200 GB one, and it is the default way to read text in Python for exactly that reason.

csv.DictReader and csv.reader stream the same way, as long as you do not wrap them in list().

Write your own streaming step with yield

def parse_events(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                yield json.loads(line)

def large_orders(events, threshold):
    for e in events:
        if e.get("amount", 0) > threshold:
            yield e

for order in large_orders(parse_events(path), 10_000):
    print(order["id"])

A function containing yield is a generator: calling it produces an object that does nothing until you loop over it, and then produces one value at a time. Nothing above holds more than one record in memory, and the three stages read as a pipeline. The generators lesson in the last module takes the mechanism apart; this is the use you will reach for first.

Two properties to know now. A generator is single-use — loop over it twice and the second loop sees nothing, with no error. And the work happens while you consume it, so an exception from line 4 million surfaces in the middle of your loop rather than at the call.

Counting without loading

total = sum(1 for _ in open(path, encoding="utf-8"))

from collections import Counter
counts = Counter(json.loads(l)["type"] for l in open(path, encoding="utf-8"))

Both use constant memory regardless of file size. sum, max, min, any, all and Counter all accept generators, which is what makes summary statistics on a huge file a one-liner.

Peeking at the first few

from itertools import islice

with open(path, encoding="utf-8") as f:
    for line in islice(f, 5):
        print(line[:200])

islice takes the first n items of any iterator without reading the rest. This is how you inspect the shape of a 40 GB file in a fraction of a second, and it is the right first move on any file you have not seen before. On the command line, head -5 file.jsonl does the same thing.

Binary files, in chunks

with open(src, "rb") as fin, open(dst, "wb") as fout:
    while chunk := fin.read(1024 * 1024):
        fout.write(transform(chunk))

A megabyte at a time. The := walrus operator assigns and tests in one expression, which is the tidy way to write a read-until-empty loop.

For hashing a large file, the same pattern feeds the hash incrementally, so you never hold more than a chunk.

Writing as you go

with open(out, "w", encoding="utf-8") as f:
    for record in transform(parse_events(path)):
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

Building the whole output in a list and writing at the end doubles your memory requirement for no benefit. Write each record as it is produced. If the job dies at 80 per cent you also keep 80 per cent of the output, which the all-at-once version does not.

When streaming is the wrong answer

If you need to sort the whole file, join it against another file, or answer questions in a different order than the data is in, streaming stops helping — those operations need the data somewhere it can be indexed.

At that point the cheapest tool is already installed:

import sqlite3

sqlite3 ships with Python, writes a single file, handles databases far larger than memory, and gives you indexes, sorting and joins in SQL. Loading a 20 GB CSV into SQLite once and querying it is often an order of magnitude faster and simpler than any amount of clever Python. It runs on a phone, needs no server, and is the most widely deployed database in the world.

The other option is pandas with chunksize, which reads a large file in blocks — covered in the next module, where the memory arithmetic gets its own treatment.