Topic 17 of 28 · Beginner
Why does a DataFrame filter not immediately process the data?
A DataFrame transformation builds a logical plan. An action such as count, show or write requests execution; creating another DataFrame does not itself materialize rows.
Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.
Detailed answer
A DataFrame transformation builds a logical plan. An action such as count, show or write requests execution; creating another DataFrame does not itself materialize rows.
Lazy planning lets Spark push filters and prune columns before reading data. A later action can execute a very different physical plan from the sequence of API calls.
Some metadata access and schema inference can still perform work. Do not interpret laziness as meaning every API invocation is entirely free.
Internal working
Catalyst resolves and optimizes a logical plan before physical operators are scheduled as tasks on partitions.
Practical scenario
A developer times only filter construction and reports an implausibly fast pipeline; time the write action and inspect the executed plan.
from pyspark.sql import SparkSession
spark=SparkSession.builder.master("local[2]").appName("lazy").getOrCreate()
df=spark.range(10).filter("id > 5")
print(df.count())
spark.stop()Common mistakes
- Timing plan construction instead of an action.
- Calling collect on a large DataFrame.
Key takeaways
- Transformations describe work.
- Actions materialize results.
- Measure the executed plan.
Follow-up challenges
- What can trigger extra schema inference work?
- How does caching change repeated actions?