Topic 12 of 28 · Advanced
Partitions, repartition & coalesce
Controlling parallelism and output file counts.
Concept
Partitions are the unit of parallelism. Too few → idle cores; too many → overhead of tiny tasks. Aim for ~100–200 MB per partition.
repartition(n or cols) does a full shuffle and can increase or decrease partitions evenly. coalesce(n) only merges partitions, avoiding a shuffle, so it's for decreasing.
spark.sql.shuffle.partitions (default 200) sets partitions after shuffles; AQE can adjust it automatically.
Code (PySpark 3.5)
print(orders.rdd.getNumPartitions())
by_city = orders.repartition(50, "city") # shuffle, co-locate city
small_out = by_city.coalesce(8) # no shuffle, fewer files
small_out.write.mode("overwrite").parquet("/lake/orders_by_city")
spark.conf.set("spark.sql.shuffle.partitions", "64")Remember
- repartition = shuffle; coalesce = merge.
- Use coalesce before writing to avoid many small files.
- Tune shuffle.partitions to data size.
Interview questions
repartition vs coalesce?
repartition does a full shuffle and balances data, for increasing or decreasing. coalesce merges partitions without a shuffle, only for decreasing, possibly unevenly.
What is the small files problem?
Thousands of tiny files slow down listing and reading. Fix with coalesce/repartition before writing or compaction jobs.
Lesson test
Take an 8-question test on “Partitions, repartition & coalesce” — concepts, output prediction, bug spotting and interview questions.