Topic 14 of 28 · Expert
Performance Tuning & Data Skew
AQE, skew handling, salting and reading the Spark UI.
Concept
Adaptive Query Execution (AQE, on by default in Spark 3) re-optimises at run time: merges small shuffle partitions, switches to broadcast joins and splits skewed partitions.
Data skew = a few keys hold most of the data, so a few tasks run much longer. Fix with AQE skew join, broadcasting, or salting the hot key.
Use the Spark UI to find slow stages, big shuffle reads/writes, spill to disk and skewed task durations.
Cheat sheet
| Symptom | Likely fix |
|---|---|
| One task much slower | Skew → salting / AQE skew join |
| Huge shuffle | Broadcast small side, filter earlier |
| Spill to disk | More partitions or executor memory |
| Many tiny tasks | Fewer shuffle partitions / AQE coalesce |
Code (PySpark 3.5)
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
# Manual salting for a hot join key
N = 10
salted_orders = orders.withColumn("salt", (F.rand() * N).cast("int"))
salted_cust = customers.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))
joined = salted_orders.join(salted_cust, ["customer_id", "salt"])Remember
- Filter and select columns as early as possible.
- Skew shows as one long task in a stage.
- Salting spreads a hot key across N partitions.
Interview questions
How do you handle data skew?
Enable AQE skew join, broadcast the small table, salt the skewed key, or process hot keys separately.
What does AQE do?
Uses runtime statistics to coalesce shuffle partitions, convert sort-merge to broadcast joins and split skewed partitions.
Lesson test
Take an 8-question test on “Performance Tuning & Data Skew” — concepts, output prediction, bug spotting and interview questions.