← PySpark roadmap

Topic 20 of 28 · Advanced

How would you diagnose a single long-running join task?

Compare task durations, shuffle read sizes and spill across the stage. A small set of large partitions suggests key skew; uniform slow tasks point to a different bottleneck.

AdvancedMid-level · 3–5 yearsPerformance

Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.

Detailed answer

Compare task durations, shuffle read sizes and spill across the stage. A small set of large partitions suggests key skew; uniform slow tasks point to a different bottleneck.

Use AQE skew handling where applicable, broadcast a genuinely small side, or split hot keys. Salting replicates the small side across salt values and distributes the matching large rows.

Salting adds complexity and can multiply data. Preserve join semantics, distinguish null keys and hot-key behavior, and validate counts before and after the change.

Internal working

Hash partitioning sends equal keys to the same partition. Increasing total partitions cannot divide a single hot key without changing the partition key or using a skew-aware strategy.

Practical scenario

One anonymous user ID represents half of a clickstream. A sort-merge join sends those records to one oversized task.

from pyspark.sql import SparkSession
spark=SparkSession.builder.master("local[2]").appName("aqe").getOrCreate()
spark.conf.set("spark.sql.adaptive.enabled","true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled","true")
print(spark.conf.get("spark.sql.adaptive.enabled"))
spark.stop()
Common mistakes
  • Increasing partitions without addressing the hot key.
  • Broadcasting a side that cannot fit on executors.
Key takeaways
  • Measure skew.
  • Choose a semantics-preserving strategy.
  • Broadcast requires a memory budget.
Follow-up challenges
  • When does salting break an aggregation?
  • How do you prove row-count equivalence?
Practise PySpark with the AI mentor

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.