← PySpark roadmap

Topic 13 of 28 · Advanced

Caching & Persistence

cache, persist storage levels and when to unpersist.

Concept

If a DataFrame is used by several actions, cache it so Spark doesn't recompute it from the source each time.

cache() = persist(MEMORY_AND_DISK) for DataFrames. Other levels include MEMORY_ONLY, DISK_ONLY and serialized variants.

Caching is lazy — it fills on the next action. Free memory with unpersist() when done.

Code (PySpark 3.5)

from pyspark import StorageLevel

clean = orders.filter("amount > 0").dropDuplicates(["order_id"])
clean.persist(StorageLevel.MEMORY_AND_DISK)
clean.count()                       # materialise cache

clean.groupBy("city").count().show()
clean.groupBy("year").sum("amount").show()

clean.unpersist()

Remember

  • Cache only data reused multiple times.
  • Cache is lazy.
  • Check the Storage tab in Spark UI.

Interview questions

cache vs persist?

cache() uses the default storage level; persist() lets you choose (memory only, disk only, serialized, replicated).

When is caching harmful?

When data is used once or is huge — it wastes executor memory, causes spills and evicts more useful data.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Caching & Persistence” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.