Threads, processes, and the lock that decides which one you need

Two kinds of slow

A program is slow in one of two ways. It is waiting — for a server, a disk, a database — or it is computing — running your own loops and arithmetic. The tools for the two are different, and using the wrong one gives you a program that is exactly as slow as before, with extra complexity.

The previous lesson handled waiting with asyncio. This lesson covers the two other tools and the fact about Python that decides between them.

Threads

A thread is a second line of execution inside the same process, sharing the same memory. Python's standard library gives you a pool of them:

from concurrent.futures import ThreadPoolExecutor

def ask(prompt):
    r = session.post(url, headers=headers, json=payload_for(prompt), timeout=60)
    r.raise_for_status()
    return r.json()

with ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(ask, prompts))

That is the whole change. ask is an ordinary synchronous function using requests; nothing became async. pool.map runs it on up to five threads at once and returns results in input order. Twenty one-second calls finish in about four seconds with five workers.

This is the right tool when the code is already synchronous and only one corner needs to overlap its waiting. It is also the right tool when you depend on a library that has no async version. max_workers is your concurrency cap, doing the job the Semaphore did in the async version.

as_completed gives results as they finish rather than in order, useful for printing progress:

from concurrent.futures import as_completed

with ThreadPoolExecutor(max_workers=5) as pool:
    futures = {pool.submit(ask, p): p for p in prompts}
    for fut in as_completed(futures):
        prompt = futures[fut]
        try:
            save(prompt, fut.result())
        except Exception as e:
            log.warning("failed %s: %s", prompt[:30], e)

fut.result() re-raises whatever exception the function raised on its thread, so errors are handled where you can see them, not lost on a background thread.

The lock

Here is the fact. CPython — the Python you installed — has a global interpreter lock, the GIL. Only one thread can execute Python bytecode at any moment. Eight threads doing arithmetic on an eight-core machine take turns on one core and finish no faster than one thread would. The lock exists because CPython's memory management was designed around it, and removing it without slowing single-threaded code has taken decades of work; a free-threaded build exists as an option in recent versions and is not yet the default.

So why do threads help with network calls? Because a thread releases the GIL while it waits. Waiting for a socket, a file, a sleep — all of these hand the lock to another thread. Threads overlap waiting perfectly and overlap computing not at all.

The same is true of C code that releases the lock on purpose. NumPy releases it inside large array operations, which is one reason NumPy on eight cores can be genuinely parallel while a Python loop over the same numbers is not.

Processes

For computing, the tool is a process pool. Each process is a separate Python interpreter with its own lock, so eight of them use eight cores.

from concurrent.futures import ProcessPoolExecutor

def score(chunk):
    return [expensive_similarity(a, b) for a, b in chunk]

if __name__ == "__main__":
    chunks = split(pairs, 8)
    with ProcessPoolExecutor(max_workers=8) as pool:
        results = list(pool.map(score, chunks))

Two costs come with it.

Nothing is shared. Every argument is serialised with Python's pickle module, sent to the worker process, and the result serialised back. Sending a 2 GB array to eight workers copies it eight times. Send small descriptions of work — a filename, a range of indices — and let each worker load what it needs. One safety note while the word is in front of you: pickle can execute code when it loads, so it is only ever safe between your own processes; never load a pickled file that came from someone else, and use JSON for anything that crosses a trust boundary.

Anything you pass must be picklable. Functions defined at module level are; lambdas, nested functions and open connections are not. The if __name__ == "__main__": guard is mandatory on macOS and Windows, where a new process re-imports your script and would otherwise start another pool inside itself, forever.

Which one

Ask what the program is doing while it is slow.

• Waiting on the network, and the code is already async or can be: asyncio.
• Waiting on the network, and the code is synchronous: ThreadPoolExecutor.
• Computing in pure Python: ProcessPoolExecutor, or rewrite the hot loop in NumPy, which is usually faster than either.
• Computing in NumPy already: it may be parallel already; measure before adding processes.

Measure. time.perf_counter() around the slow part, run once with one worker and once with several. If more workers do not help, you have the wrong tool, and the number tells you before you have restructured the program.

The shared-state trap

Threads share memory, and that cuts both ways. Two threads appending to the same list is fine, because list.append is one bytecode operation and cannot be interrupted midway. Two threads doing counter += 1 is not: read, add, write are three steps, and a thread can be switched out between them, so counts go missing. Guard shared mutation with a lock, or, more simply, have each thread return its result and combine them in the main thread afterwards, as pool.map already does.

A Session from requests is safe to share across threads for ordinary use; the pool inside it is designed for that. A sqlite3 connection is not; open one per thread.

Try this now

Write a pure-Python function that sums the squares of a million numbers. Time it once, then with ThreadPoolExecutor(4) splitting the range four ways, then with ProcessPoolExecutor(4). Then do the same sum with numpy.arange(1_000_000) ** 2 and .sum(). Write the four numbers down; they are the whole lesson.