4 · Advanced · lesson 16 of 20

Window Functions

Row-by-row calculations that need context — running totals, ranks, lags.

Python
from pyspark.sql import Window
from pyspark.sql import functions as F

w = Window.partitionBy("customer_id").orderBy("created_at")

enriched = (df
    .withColumn("order_no", F.row_number().over(w))
    .withColumn("prev_amount", F.lag("amount").over(w))
    .withColumn(
        "rolling_7",
        F.sum("amount").over(w.rowsBetween(-6, 0)),
    )
)
  • ▸row_number, rank, dense_rank — ordering.
  • ▸lag, lead — look at the previous / next row.
  • ▸sum / avg / min / max over a Window — rolling stats.
  • ▸rowsBetween(-6, 0) — the last 7 rows including this one.
Key takeaways
  • ✓Window = partitionBy (group) + orderBy (sort inside group) + optional frame.
  • ✓One shuffle per unique Window spec — reuse Window objects where possible.