← PySpark roadmap

Topic 6 of 28 · Intermediate

GroupBy & Aggregations

groupBy, agg, pivot and handling nulls.

Concept

groupBy(...).agg(...) computes summaries per group using functions like sum, avg, count, countDistinct, max and collect_list.

pivot turns distinct values into columns — pass the list of values to avoid an extra scan.

Handle nulls with fillna, dropna, coalesce and na.replace before aggregating.

Code (PySpark 3.5)

summary = (orders
    .groupBy("city")
    .agg(F.count("*").alias("orders"),
         F.sum("amount").alias("revenue"),
         F.round(F.avg("amount"), 2).alias("avg_amount"),
         F.countDistinct("customer_id").alias("customers"))
    .filter(F.col("revenue") > 10000)
    .orderBy(F.desc("revenue")))

monthly = (orders
    .withColumn("month", F.date_format("order_date", "yyyy-MM"))
    .groupBy("city").pivot("month", ["2026-01", "2026-02"]).sum("amount"))

Remember

  • Alias every aggregate.
  • countDistinct is expensive; approx_count_distinct is cheaper.
  • Provide pivot values explicitly.

Interview questions

reduceByKey vs groupByKey?

reduceByKey combines values on each partition before shuffling, moving far less data; groupByKey ships every value across the network.

How do you get distinct count quickly on huge data?

approx_count_distinct (HyperLogLog) with an acceptable error margin.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “GroupBy & Aggregations” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.