Topic 13 of 28 · Advanced
Caching & Persistence
cache, persist storage levels and when to unpersist.
Concept
If a DataFrame is used by several actions, cache it so Spark doesn't recompute it from the source each time.
cache() = persist(MEMORY_AND_DISK) for DataFrames. Other levels include MEMORY_ONLY, DISK_ONLY and serialized variants.
Caching is lazy — it fills on the next action. Free memory with unpersist() when done.
Code (PySpark 3.5)
from pyspark import StorageLevel
clean = orders.filter("amount > 0").dropDuplicates(["order_id"])
clean.persist(StorageLevel.MEMORY_AND_DISK)
clean.count() # materialise cache
clean.groupBy("city").count().show()
clean.groupBy("year").sum("amount").show()
clean.unpersist()Remember
- Cache only data reused multiple times.
- Cache is lazy.
- Check the Storage tab in Spark UI.
Interview questions
cache vs persist?
cache() uses the default storage level; persist() lets you choose (memory only, disk only, serialized, replicated).
When is caching harmful?
When data is used once or is huge — it wastes executor memory, causes spills and evicts more useful data.
Lesson test
Take an 8-question test on “Caching & Persistence” — concepts, output prediction, bug spotting and interview questions.