1 · Foundations · lesson 4 of 20

Transformations & Actions

Why nothing runs until you ask for a result — and why that's a feature.

Every Spark operation is either a transformation (returns a new DataFrame, does no work yet) or an action (triggers computation and returns a value to the Driver).

Python
df = spark.read.parquet("/data/orders")

# Transformations — LAZY, build a plan
filtered = df.filter(df.amount > 100)
by_country = filtered.groupBy("country").sum("amount")

# Action — triggers execution
by_country.show()          # runs the plan
by_country.count()         # runs it AGAIN unless cached
  • ▸Transformations: select, filter, groupBy, join, withColumn, agg…
  • ▸Actions: show, count, collect, take, write, foreach.
  • ▸Spark inspects the whole plan before running — it can push filters down and prune columns.
WATCH OUT
Every action re-runs the plan from scratch. Cache or write intermediate results if you'll use them more than once.
Key takeaways
  • ✓Transformations are lazy — they only build the plan.
  • ✓Actions trigger execution and pull a result back to the Driver.
  • ✓Lazy evaluation is what lets Catalyst optimize the whole query.