← PySpark roadmap

Topic 12 of 28 · Advanced

Partitions, repartition & coalesce

Controlling parallelism and output file counts.

Concept

Partitions are the unit of parallelism. Too few → idle cores; too many → overhead of tiny tasks. Aim for ~100–200 MB per partition.

repartition(n or cols) does a full shuffle and can increase or decrease partitions evenly. coalesce(n) only merges partitions, avoiding a shuffle, so it's for decreasing.

spark.sql.shuffle.partitions (default 200) sets partitions after shuffles; AQE can adjust it automatically.

Code (PySpark 3.5)

print(orders.rdd.getNumPartitions())

by_city = orders.repartition(50, "city")      # shuffle, co-locate city
small_out = by_city.coalesce(8)               # no shuffle, fewer files
small_out.write.mode("overwrite").parquet("/lake/orders_by_city")

spark.conf.set("spark.sql.shuffle.partitions", "64")

Remember

  • repartition = shuffle; coalesce = merge.
  • Use coalesce before writing to avoid many small files.
  • Tune shuffle.partitions to data size.

Interview questions

repartition vs coalesce?

repartition does a full shuffle and balances data, for increasing or decreasing. coalesce merges partitions without a shuffle, only for decreasing, possibly unevenly.

What is the small files problem?

Thousands of tiny files slow down listing and reading. Fix with coalesce/repartition before writing or compaction jobs.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Partitions, repartition & coalesce” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.