Generators: Functions That Pause
In short: write an ordinary function, put yield where you'd return a
value, and Python builds lesson 01's two-method, one-exception iterator for
you. Calling a generator function runs none of its body — it returns a
generator object immediately, and the body only starts at the first
next(). Each next() runs to the next yield, hands back that value, and
parks the frame: every local variable survives, waiting for the call
after. A generator is still a stream, not a container — walk it twice and
the second walk is empty. The piece you'll build is Quarry's line feed:
files are read in fixed-size chunks, and a chunk boundary can land in the
middle of a line.
Rewrite Quarry's row cursor in five lines
Lesson 01's row cursor took two classes, a position attribute, and a
hand-raised StopIteration. A generator function — a function whose
body contains yield — does the same job in five lines:
No pos, no __next__, no raise. The for loop inside the function is
the position, and when the function finishes, Python raises
StopIteration for you. The mistake almost everyone makes on day one:
assuming the function's body runs when you call it.
def feed():
print("open", end=" ")
yield 1
print("close", end=" ")
g = feed()
print("made", end=" ")
print(next(g))type(g).__name__ is "generator", and iter(g) is g is True — a
generator is an ordinary iterator of the kind you wrote by hand in lesson
01, which is why it's a stream: walk it twice and the second walk is empty.
Watch a generator's frame pause and resume
When you call an ordinary function, Python makes a frame — the call's
local variables and the line it's on — pushes it on the call stack, and
discards it on return. A generator's frame is different: at each yield it
comes off the stack but is kept, every local intact. The next
next() puts that same frame back on, resuming right after the yield.
Watch the stack as a loop pulls three lines from a numbering generator:
A generator's frame leaves the call stack at every yield and comes back at the next next(). Its locals survive in between.
next(g) pushes the generator's frame, and it runs until the first yield. Stack, 2 items. Top is numbered() · n=1, line='a'.
The same thing, told as a conversation between the loop and the generator:
Now step through the real thing. Notice how the trace jumps into the
generator's body at each next() and back out at each yield, and that
n still holds its old value every time it jumps back in:
def numbered(lines):
n = 0
for line in lines:
n += 1
yield f"{n}:{line}"
g = numbered(["a", "b", "c"])
next(g)
print(next(g), end=" ")
print(list(g))inspect.getgeneratorstate(g) confirms the three states directly: freshly
made (GEN_CREATED), parked at a yield (GEN_SUSPENDED), and finished
(GEN_CLOSED) — worth a look if you want to see the lifecycle by name.
Read a file in chunks without loading it
Quarry never reads a 200 MB file with .read() — that puts the whole file
in memory. It reads a fixed-size chunk at a time, and a generator is the
natural shape:
Look at where the boundaries fall: 'boot o', 'k\ndisk'. A chunk has no
idea where a line begins or ends, so the feed has to carry each chunk's
unfinished tail forward and glue it onto the next one — only complete lines
leave. The carry never grows past one chunk plus one line, however big the
file gets, which is the whole point of streaming: memory stays flat while
the file grows.
Trap: inside a generator, return means "stop right now" — it raises
StopIteration immediately, which is an easy way to break a generator with
one wrong word:
Run it and look closely at what list() received. Then change one word so every non-blank line comes through.
Whether a function is a generator is decided by whether yield appears
anywhere in its body — settled when the function is compiled, not when
it runs. One stray return where you meant yield turns a generator back
into an ordinary function, silently.
Quick practice
import sys
small = (n for n in range(10))
huge = (n for n in range(10_000_000))
print(sys.getsizeof(small) == sys.getsizeof(huge))def once():
yield "a"
yield "b"
g = once()
print(list(g), list(g))def first_word(lines):
for line in lines:
if line:
yield line.split()[0]
words = first_word(["boot ok", "", "disk warm"])
print(next(words), next(words))🎯 Your turn
Write the generator lines_from_chunks(chunks). It takes any iterable of
text chunks and yields every complete line without its "\n". It
carries the unfinished tail from one chunk to the next, and at the end
yields whatever is left over if it isn't empty.
chunks(text, size) is written for you. It stands in for reading a file
size characters at a time. Then read_lines(text, size) returns
list(lines_from_chunks(chunks(text, size))):
read_lines("boot ok\nwarm\n", 4)→["boot ok", "warm"]read_lines("a\nbb\nccc", 2)→["a", "bb", "ccc"](no final newline)read_lines("one\n\nthree\n", 1)→["one", "", "three"](a blank line is still a line)
Hint: carry += chunk, then *done, carry = carry.split("\n").
Everything in done is a complete line.
Press ▶ Run, then ✓ Check. ✅