← PySpark roadmap

Topic 5 of 28 · Intermediate

Lazy Evaluation, Actions & DAG

Transformations vs actions, jobs, stages and tasks.

Concept

Transformations (filter, select, join) only build a plan. Nothing runs until an action (show, count, collect, write) is called.

Each action creates a job. The job's DAG is cut into stages at shuffle boundaries, and each stage runs one task per partition.

Narrow transformations (map, filter) don't move data between partitions; wide ones (groupBy, join, distinct) cause a shuffle.

Cheat sheet

NarrowWide (shuffle)
select, filtergroupBy, agg
withColumnjoin (non-broadcast)
uniondistinct, orderBy
coalescerepartition

Code (PySpark 3.5)

big = orders.filter("amount > 1000")    # nothing runs yet
by_city = big.groupBy("city").count()    # still nothing

by_city.explain(True)   # see the logical and physical plans
by_city.show()          # ACTION -> job runs

Remember

  • No action, no execution.
  • Shuffle = stage boundary.
  • Never collect() a large DataFrame to the driver.

Interview questions

Why is Spark lazy?

So it can see the whole plan and optimise it — combine steps, push filters down, prune columns — before running.

Job vs stage vs task?

An action triggers a job; a job splits into stages at shuffles; each stage runs one task per partition.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Lazy Evaluation, Actions & DAG” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.