The iterator protocol: what for actually does, and why a generator is empty the second time

Two calls under every loop

for item in things:
    ...

is shorthand for:

it = iter(things)
while True:
    try:
        item = next(it)
    except StopIteration:
        break
    ...

iter(things) calls things.__iter__() and gets back an iterator: an object with a __next__ method. next(it) calls it. When there is nothing left, __next__ raises StopIteration and the loop ends. That is the entire protocol, and every for, every comprehension, every sum, list, max, zip and enumerate runs on it.

Two roles are in play. An iterable is anything iter() accepts — a list, a string, a dict, a file, your Conversation with its __iter__. An iterator is the thing that hands out items one at a time and remembers where it is. The distinction seems academic until it bites.

Lists give you a fresh iterator; generators are the iterator

nums = [1, 2, 3]
print(sum(nums), sum(nums))       # 6 6

gen = (n for n in [1, 2, 3])
print(sum(gen), sum(gen))         # 6 0

Each iter(nums) returns a new iterator starting at position 0, so a list can be walked any number of times. A generator object is its own iterator: iter(gen) returns gen itself, and it remembers that it already reached the end. The second sum asks for next, gets StopIteration immediately, and returns 0.

No error, no warning. The second pass is just empty. This is the single most common surprise for people who have started using generators — and you have been using them since module 5's streaming and module 6's pagination. The pattern that triggers it: count something, then process it, over the same generator. Or pass a generator to a function that iterates it twice internally, which zip does not but a few library functions do.

The fix is to decide what you want. If you need to walk the data twice, materialise it once: lines = list(gen). If the data is too large for that — the reason you used a generator — restructure so that one pass does both jobs, or create the generator twice by calling the function that makes it twice.

Writing an iterator by hand

You rarely need to, because yield does it, but seeing one makes the protocol concrete:

class Countdown:
    def __init__(self, n):
        self.n = n
    def __iter__(self):
        return self
    def __next__(self):
        if self.n <= 0:
            raise StopIteration
        self.n -= 1
        return self.n + 1

__iter__ returning self is what makes it an iterator rather than merely iterable — and is what makes it single-use, for the same reason a generator is. A generator function is this class written as a function:

def countdown(n):
    while n > 0:
        yield n
        n -= 1

Each next() runs the function up to the next yield, hands out the value, and freezes the frame — local variables, position, everything — until the following next(). When the function returns, StopIteration is raised for you. Laziness comes for free: nothing after the current yield has run yet.

next() with a default, and peeking

first = next(iter(things), None)

gives you the first item or None without a loop and without IndexError. It works on any iterable, including a generator of unknown length. To look at the first few of a large stream without consuming all of it:

from itertools import islice
head = list(islice(gen, 5))

islice takes the first five and stops; the rest of gen is still there for a later loop. That is one of the few ways to sample a generator without materialising it, and it is what the "peek at the first rows" idiom in module 5 was doing.

itertools pieces you will use

• chain(a, b) — iterate a then b as one sequence, without building a combined list.
• batched(it, 20) (3.12+) — groups of twenty, the natural shape for sending twenty prompts per request or per thread. Before 3.12, write a small yield loop that fills a list and yields it when full.
• groupby(sorted_items, key=...) — consecutive runs with the same key. It only groups adjacent items, so sort first or it silently produces many small groups.
• count() — infinite integers, for numbering a stream of unknown length alongside zip.

Every one of these is lazy. A pipeline of generators — read lines, strip them, filter, batch, send — processes one item at a time end to end, and its memory use does not grow with the file.

zip and unequal lengths

zip(a, b) stops at the shorter. If a has 100 items and b has 99 because of an off-by-one upstream, zip drops the last silently. zip(a, b, strict=True) (3.10+) raises instead. Use it whenever the two sequences are supposed to match, such as prompts and their results.

Where this leaves you

You now know what for does, why a file can be looped once, why iter_lines in module 6 could be handed to any function that takes an iterable, and why results = list(gen) appears at the end of so many pipelines: it is the moment the data stops being lazy and becomes something you can walk twice.

Try this now

Write a generator that yields chunks of 100 items from any iterable. Feed it a range(1050), check that the last chunk has 50, then consume it with sum(len(c) for c in chunks) and try to loop over chunks again.