Topic 20 of 28 · Advanced
How would you diagnose a single long-running join task?
Compare task durations, shuffle read sizes and spill across the stage. A small set of large partitions suggests key skew; uniform slow tasks point to a different bottleneck.
Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.
Detailed answer
Compare task durations, shuffle read sizes and spill across the stage. A small set of large partitions suggests key skew; uniform slow tasks point to a different bottleneck.
Use AQE skew handling where applicable, broadcast a genuinely small side, or split hot keys. Salting replicates the small side across salt values and distributes the matching large rows.
Salting adds complexity and can multiply data. Preserve join semantics, distinguish null keys and hot-key behavior, and validate counts before and after the change.
Internal working
Hash partitioning sends equal keys to the same partition. Increasing total partitions cannot divide a single hot key without changing the partition key or using a skew-aware strategy.
Practical scenario
One anonymous user ID represents half of a clickstream. A sort-merge join sends those records to one oversized task.
from pyspark.sql import SparkSession
spark=SparkSession.builder.master("local[2]").appName("aqe").getOrCreate()
spark.conf.set("spark.sql.adaptive.enabled","true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled","true")
print(spark.conf.get("spark.sql.adaptive.enabled"))
spark.stop()Common mistakes
- Increasing partitions without addressing the hot key.
- Broadcasting a side that cannot fit on executors.
Key takeaways
- Measure skew.
- Choose a semantics-preserving strategy.
- Broadcast requires a memory budget.
Follow-up challenges
- When does salting break an aggregation?
- How do you prove row-count equivalence?