← PySpark roadmap

Topic 18 of 28 · Beginner

How do jobs, stages and tasks relate to a shuffle?

An action normally creates a job. Spark divides its execution into stages separated by shuffle dependencies; each stage processes partitions through tasks.

BasicEntry level · 0–1 yearsExecution

Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.

Detailed answer

An action normally creates a job. Spark divides its execution into stages separated by shuffle dependencies; each stage processes partitions through tasks.

Narrow dependencies permit pipelined work on a partition, while grouping or many joins redistribute records by key. A broadcast join can avoid shuffling the large side.

AQE can change the executed physical plan and partition count. Avoid asserting that every action produces exactly one stage or every join always shuffles both sides.

Internal working

Map outputs are exchanged between partitions at a shuffle boundary. Network transfer, sorting, spill and skew can dominate execution time.

Practical scenario

A small grouping operation takes longer than a large filter because grouping introduces an exchange and an uneven hot-key partition.

from pyspark.sql import SparkSession
spark=SparkSession.builder.master("local[2]").appName("stages").getOrCreate()
spark.range(20).selectExpr("id % 3 AS key").groupBy("key").count().explain()
spark.stop()
Common mistakes
  • Equating API calls with stages.
  • Assuming repartition is free.
Key takeaways
  • Inspect shuffle dependencies.
  • Tasks process partitions.
  • AQE can alter the plan.
Follow-up challenges
  • How does map-side aggregation reduce traffic?
  • What does the Spark UI reveal about skew?
Practise PySpark with the AI mentor

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.