Topic 2 of 28 · Beginner
RDDs vs DataFrames
Low-level RDDs, structured DataFrames and the Catalyst optimizer.
Concept
An RDD (Resilient Distributed Dataset) is an immutable, partitioned collection of objects. It's flexible but Spark can't optimise its Python lambdas.
A DataFrame is a distributed table with named columns and a schema. Spark's Catalyst optimizer rewrites DataFrame plans and Tungsten executes them efficiently.
Use DataFrames (or Spark SQL) for nearly everything; drop to RDDs only for unusual low-level needs.
Code (PySpark 3.5)
rdd = spark.sparkContext.parallelize([("Asha", 90), ("Ravi", 70)])
print(rdd.map(lambda x: x[1]).sum())
df = spark.createDataFrame([("Asha", 90), ("Ravi", 70)], ["name", "score"])
df.printSchema()
df.filter(df.score > 75).show()Remember
- DataFrames are optimised; RDD lambdas are not.
- Both are immutable and lazily evaluated.
- Resilient = rebuilt from lineage on failure.
Interview questions
Why are DataFrames faster than RDDs in Python?
DataFrame operations run inside the JVM, optimised by Catalyst; RDD Python lambdas must serialize data to Python workers.
What does 'resilient' mean?
Lost partitions are recomputed from the lineage (the recorded transformations) instead of being replicated.
Lesson test
Take an 8-question test on “RDDs vs DataFrames” — concepts, output prediction, bug spotting and interview questions.