← Python roadmap

How does a generator retain state without storing the entire result?

IntermediateJunior · 1–3 yearsIteration

Python 3.10+; standard library only. GIL discussion refers to conventional CPython builds, not optional free-threaded builds.

Detailed answer

Calling a generator function creates an iterator without running its body immediately. next() resumes execution until yield, which returns a value and suspends local state.

Generators permit streaming over large inputs using bounded working memory when neither the producer nor consumer accumulates results. Converting to list defeats this advantage.

Generators are normally single-use. Handle exhaustion, resource cleanup and exceptions; use a context manager around an open input file rather than relying on garbage collection.

Internal working

A suspended generator retains its execution frame and local variables. yield from delegates iteration and propagates send/throw/return behavior, not merely values.

Practical scenario

Process a large CSV one record at a time and aggregate counts without loading the entire file.

def squares(values):
    for x in values:
        yield x * x
g = squares(range(3))
print(next(g), list(g)) # 0 [1,4]
Common mistakes
  • Calling list on an unbounded iterator.
  • Reusing an exhausted generator.
Key takeaways
  • Execution starts on iteration.
  • Suspension retains local state.
  • Streaming requires a bounded consumer too.
Follow-up challenges
  • How would you close a generator holding a resource?
  • How does yield from differ from a simple for loop?