← PySpark roadmap

Topic 14 of 28 · Expert

Performance Tuning & Data Skew

AQE, skew handling, salting and reading the Spark UI.

Concept

Adaptive Query Execution (AQE, on by default in Spark 3) re-optimises at run time: merges small shuffle partitions, switches to broadcast joins and splits skewed partitions.

Data skew = a few keys hold most of the data, so a few tasks run much longer. Fix with AQE skew join, broadcasting, or salting the hot key.

Use the Spark UI to find slow stages, big shuffle reads/writes, spill to disk and skewed task durations.

Cheat sheet

SymptomLikely fix
One task much slowerSkew → salting / AQE skew join
Huge shuffleBroadcast small side, filter earlier
Spill to diskMore partitions or executor memory
Many tiny tasksFewer shuffle partitions / AQE coalesce

Code (PySpark 3.5)

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

# Manual salting for a hot join key
N = 10
salted_orders = orders.withColumn("salt", (F.rand() * N).cast("int"))
salted_cust = customers.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))
joined = salted_orders.join(salted_cust, ["customer_id", "salt"])

Remember

  • Filter and select columns as early as possible.
  • Skew shows as one long task in a stage.
  • Salting spreads a hot key across N partitions.

Interview questions

How do you handle data skew?

Enable AQE skew join, broadcast the small table, salt the skewed key, or process hot keys separately.

What does AQE do?

Uses runtime statistics to coalesce shuffle partitions, convert sort-merge to broadcast joins and split skewed partitions.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Performance Tuning & Data Skew” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.