Topic 1 of 28 · Beginner
What is Spark?
Distributed processing, driver, executors and cluster managers.
Concept
Apache Spark is an engine for processing large data in parallel across many machines, keeping data in memory where possible. PySpark is its Python API.
A Spark application has one driver (plans the work, runs your main code) and many executors (run tasks on partitions of data). A cluster manager (YARN, Kubernetes, Standalone) gives it resources.
Spark is far faster than MapReduce for iterative work because it avoids writing intermediate results to disk.
Cheat sheet
| Component | Role |
|---|---|
| Driver | Builds the plan, schedules tasks |
| Executor | Runs tasks, caches data |
| Cluster manager | Allocates CPU/memory |
| Task | Work on one partition |
Code (PySpark 3.5)
from pyspark.sql import SparkSession
spark = (SparkSession.builder
.appName("aicodedost")
.master("local[*]")
.getOrCreate())
print(spark.version)
spark.range(5).show()Remember
- SparkSession is the single entry point.
- One task per partition.
- local[*] uses all cores on your machine.
Interview questions
Spark vs Hadoop MapReduce?
Spark processes in memory with a DAG of stages; MapReduce writes to disk between each map and reduce step, so Spark is much faster for multi-step jobs.
What does the driver do?
Runs main(), creates SparkSession, converts code into a DAG, splits it into stages/tasks and schedules them on executors.
Lesson test
Take an 8-question test on “What is Spark?” — concepts, output prediction, bug spotting and interview questions.