Topic 6 of 28 · Intermediate
GroupBy & Aggregations
groupBy, agg, pivot and handling nulls.
Concept
groupBy(...).agg(...) computes summaries per group using functions like sum, avg, count, countDistinct, max and collect_list.
pivot turns distinct values into columns — pass the list of values to avoid an extra scan.
Handle nulls with fillna, dropna, coalesce and na.replace before aggregating.
Code (PySpark 3.5)
summary = (orders
.groupBy("city")
.agg(F.count("*").alias("orders"),
F.sum("amount").alias("revenue"),
F.round(F.avg("amount"), 2).alias("avg_amount"),
F.countDistinct("customer_id").alias("customers"))
.filter(F.col("revenue") > 10000)
.orderBy(F.desc("revenue")))
monthly = (orders
.withColumn("month", F.date_format("order_date", "yyyy-MM"))
.groupBy("city").pivot("month", ["2026-01", "2026-02"]).sum("amount"))Remember
- Alias every aggregate.
- countDistinct is expensive; approx_count_distinct is cheaper.
- Provide pivot values explicitly.
Interview questions
reduceByKey vs groupByKey?
reduceByKey combines values on each partition before shuffling, moving far less data; groupByKey ships every value across the network.
How do you get distinct count quickly on huge data?
approx_count_distinct (HyperLogLog) with an acceptable error margin.
Lesson test
Take an 8-question test on “GroupBy & Aggregations” — concepts, output prediction, bug spotting and interview questions.