Topic 5 of 28 · Intermediate
Lazy Evaluation, Actions & DAG
Transformations vs actions, jobs, stages and tasks.
Concept
Transformations (filter, select, join) only build a plan. Nothing runs until an action (show, count, collect, write) is called.
Each action creates a job. The job's DAG is cut into stages at shuffle boundaries, and each stage runs one task per partition.
Narrow transformations (map, filter) don't move data between partitions; wide ones (groupBy, join, distinct) cause a shuffle.
Cheat sheet
| Narrow | Wide (shuffle) |
|---|---|
| select, filter | groupBy, agg |
| withColumn | join (non-broadcast) |
| union | distinct, orderBy |
| coalesce | repartition |
Code (PySpark 3.5)
big = orders.filter("amount > 1000") # nothing runs yet
by_city = big.groupBy("city").count() # still nothing
by_city.explain(True) # see the logical and physical plans
by_city.show() # ACTION -> job runsRemember
- No action, no execution.
- Shuffle = stage boundary.
- Never collect() a large DataFrame to the driver.
Interview questions
Why is Spark lazy?
So it can see the whole plan and optimise it — combine steps, push filters down, prune columns — before running.
Job vs stage vs task?
An action triggers a job; a job splits into stages at shuffles; each stage runs one task per partition.
Lesson test
Take an 8-question test on “Lazy Evaluation, Actions & DAG” — concepts, output prediction, bug spotting and interview questions.