Topic 18 of 28 · Beginner
How do jobs, stages and tasks relate to a shuffle?
An action normally creates a job. Spark divides its execution into stages separated by shuffle dependencies; each stage processes partitions through tasks.
Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.
Detailed answer
An action normally creates a job. Spark divides its execution into stages separated by shuffle dependencies; each stage processes partitions through tasks.
Narrow dependencies permit pipelined work on a partition, while grouping or many joins redistribute records by key. A broadcast join can avoid shuffling the large side.
AQE can change the executed physical plan and partition count. Avoid asserting that every action produces exactly one stage or every join always shuffles both sides.
Internal working
Map outputs are exchanged between partitions at a shuffle boundary. Network transfer, sorting, spill and skew can dominate execution time.
Practical scenario
A small grouping operation takes longer than a large filter because grouping introduces an exchange and an uneven hot-key partition.
from pyspark.sql import SparkSession
spark=SparkSession.builder.master("local[2]").appName("stages").getOrCreate()
spark.range(20).selectExpr("id % 3 AS key").groupBy("key").count().explain()
spark.stop()Common mistakes
- Equating API calls with stages.
- Assuming repartition is free.
Key takeaways
- Inspect shuffle dependencies.
- Tasks process partitions.
- AQE can alter the plan.
Follow-up challenges
- How does map-side aggregation reduce traffic?
- What does the Spark UI reveal about skew?