PySpark Tutorial
Learn PySpark from Spark basics and DataFrames to joins, window functions, partitioning, performance tuning and Delta Lake. Each topic includes code, key points and data engineering interview questions.
28 topics · Beginner → Expert · concept, code, interview questions and a test in every topic
Beginner lessons
0/4 · 0%
- 1What is Spark?Distributed processing, driver, executors and cluster managers.
- 2RDDs vs DataFramesLow-level RDDs, structured DataFrames and the Catalyst optimizer.
- 3Reading & Writing DataCSV, JSON, Parquet, schemas and save modes.
- 4Selecting, Filtering & Columnsselect, withColumn, filter, when and built-in functions.
Intermediate lessons
0/4 · 0%
- 5Lazy Evaluation, Actions & DAGTransformations vs actions, jobs, stages and tasks.
- 6GroupBy & AggregationsgroupBy, agg, pivot and handling nulls.
- 7Functions Reference: Column & Row TransformationsThe built-in column functions every data engineer uses, grouped by purpose.
- 9JoinsJoin types, broadcast joins and duplicate column issues.
Advanced lessons
0/5 · 0%
- 8Functions Reference: Complex Types, Aggregations & WindowsArrays, maps, structs and JSON, plus aggregate and window functions for real pipelines.
- 10Window Functionsrow_number, rank, lag/lead and running totals.
- 11Spark SQL & UDFsTemp views, SQL queries and when (not) to write UDFs.
- 12Partitions, repartition & coalesceControlling parallelism and output file counts.
- 13Caching & Persistencecache, persist storage levels and when to unpersist.
Expert lessons
0/3 · 0%
Interview preparation
Module — Core Technical Interview Q&A (Top Questions & Answers)
6 questions · 0 completed · 0%
Module — Core Technical Interview Q&A (Top Questions & Answers)
6 questions · 0 completed · 0%
Beginner · Fundamentals
Why does a DataFrame filter not immediately process the data?
A DataFrame transformation builds a logical plan. An action such as count, show or write requests execution; creating another DataFrame does not itself materialize rows.
Basic · Execution
How do jobs, stages and tasks relate to a shuffle?
An action normally creates a job. Spark divides its execution into stages separated by shuffle dependencies; each stage processes partitions through tasks.
Intermediate · Data quality
Why is dropDuplicates not enough to select the latest customer record?
dropDuplicates on a key keeps a representative row, not the row with the latest timestamp. Its selected row is not a documented ordering guarantee.
Advanced · Performance
How would you diagnose a single long-running join task?
Compare task durations, shuffle read sizes and spill across the stage. A small set of large partitions suggests key skew; uniform slow tasks point to a different bottleneck.
Expert · Streaming
Does a watermark guarantee all late data is processed exactly once?
A watermark expresses an event-time lateness boundary used by stateful operators to finalize windows and evict state. It is not a blanket exactly-once guarantee.
Architect / Specialist · Pipeline architecture
How would you make a large incremental pipeline restartable and auditable?
Define source offsets or versions, target keys, a deterministic conflict rule and commit boundaries. A restart must repeat or resume work without creating different business results.
Module — Coding Interview Problems & Solutions (Hands-on)
6 questions · 0 completed · 0%
Module — Coding Interview Problems & Solutions (Hands-on)
6 questions · 0 completed · 0%
Beginner · DataFrames
Filter positive non-null transaction amounts
Keep rows with amount strictly greater than zero, preserving id and amount.
Basic · Aggregation
Count whitespace-separated words with DataFrame expressions
Count case-sensitive whitespace-separated nonempty words across non-null input lines.
Intermediate · Windows
Select the highest customer version with an explicit tie-breaker
Keep the highest version for each customer; break same-version ties by lexicographically smallest non-null city.
Advanced · Joins
Enrich orders with a small unique customer dimension
Left-join orders to a small customer dimension without losing unmatched orders.
Expert · Window frames
Compute a rolling mean of the last three observations per entity
Compute the mean over the current and preceding two rows per id ordered by unique day.
Architect / Specialist · Clickstream processing
Assign per-user batch sessions with a strict inactivity gap
Create a new per-user session at the first event or when the gap from the previous event exceeds ten units.
A–Z index
- Assign per-user batch sessions with a strict inactivity gap
- Caching & Persistence
- Compute a rolling mean of the last three observations per entity
- Count whitespace-separated words with DataFrame expressions
- Delta Lake & MERGE
- Does a watermark guarantee all late data is processed exactly once?
- Enrich orders with a small unique customer dimension
- Filter positive non-null transaction amounts
- Functions Reference: Column & Row Transformations
- Functions Reference: Complex Types, Aggregations & Windows
- GroupBy & Aggregations
- How do jobs, stages and tasks relate to a shuffle?
- How would you diagnose a single long-running join task?
- How would you make a large incremental pipeline restartable and auditable?
- Joins
- Lazy Evaluation, Actions & DAG
- Partitions, repartition & coalesce
- Performance Tuning & Data Skew
- RDDs vs DataFrames
- Reading & Writing Data
- Select the highest customer version with an explicit tie-breaker
- Selecting, Filtering & Columns
- Spark SQL & UDFs
- Structured Streaming
- What is Spark?
- Why does a DataFrame filter not immediately process the data?
- Why is dropDuplicates not enough to select the latest customer record?
- Window Functions
Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.