← PySpark roadmap

Topic 2 of 28 · Beginner

RDDs vs DataFrames

Low-level RDDs, structured DataFrames and the Catalyst optimizer.

Concept

An RDD (Resilient Distributed Dataset) is an immutable, partitioned collection of objects. It's flexible but Spark can't optimise its Python lambdas.

A DataFrame is a distributed table with named columns and a schema. Spark's Catalyst optimizer rewrites DataFrame plans and Tungsten executes them efficiently.

Use DataFrames (or Spark SQL) for nearly everything; drop to RDDs only for unusual low-level needs.

Code (PySpark 3.5)

rdd = spark.sparkContext.parallelize([("Asha", 90), ("Ravi", 70)])
print(rdd.map(lambda x: x[1]).sum())

df = spark.createDataFrame([("Asha", 90), ("Ravi", 70)], ["name", "score"])
df.printSchema()
df.filter(df.score > 75).show()

Remember

  • DataFrames are optimised; RDD lambdas are not.
  • Both are immutable and lazily evaluated.
  • Resilient = rebuilt from lineage on failure.

Interview questions

Why are DataFrames faster than RDDs in Python?

DataFrame operations run inside the JVM, optimised by Catalyst; RDD Python lambdas must serialize data to Python workers.

What does 'resilient' mean?

Lost partitions are recomputed from the lineage (the recorded transformations) instead of being replicated.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “RDDs vs DataFrames” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.