← PySpark roadmap

Topic 1 of 28 · Beginner

What is Spark?

Distributed processing, driver, executors and cluster managers.

Concept

Apache Spark is an engine for processing large data in parallel across many machines, keeping data in memory where possible. PySpark is its Python API.

A Spark application has one driver (plans the work, runs your main code) and many executors (run tasks on partitions of data). A cluster manager (YARN, Kubernetes, Standalone) gives it resources.

Spark is far faster than MapReduce for iterative work because it avoids writing intermediate results to disk.

Cheat sheet

ComponentRole
DriverBuilds the plan, schedules tasks
ExecutorRuns tasks, caches data
Cluster managerAllocates CPU/memory
TaskWork on one partition

Code (PySpark 3.5)

from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .appName("aicodedost")
         .master("local[*]")
         .getOrCreate())

print(spark.version)
spark.range(5).show()

Remember

  • SparkSession is the single entry point.
  • One task per partition.
  • local[*] uses all cores on your machine.

Interview questions

Spark vs Hadoop MapReduce?

Spark processes in memory with a DAG of stages; MapReduce writes to disk between each map and reduce step, so Spark is much faster for multi-step jobs.

What does the driver do?

Runs main(), creates SparkSession, converts code into a DAG, splits it into stages/tasks and schedules them on executors.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “What is Spark?” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.