← All tutorials

PySpark Tutorial

Learn PySpark from Spark basics and DataFrames to joins, window functions, partitioning, performance tuning and Delta Lake. Each topic includes code, key points and data engineering interview questions.

28 topics · Beginner → Expert · concept, code, interview questions and a test in every topic

Your progress0%

Beginner lessons

0/4 · 0%
  1. 1What is Spark?Distributed processing, driver, executors and cluster managers.
  2. 2RDDs vs DataFramesLow-level RDDs, structured DataFrames and the Catalyst optimizer.
  3. 3Reading & Writing DataCSV, JSON, Parquet, schemas and save modes.
  4. 4Selecting, Filtering & Columnsselect, withColumn, filter, when and built-in functions.

Intermediate lessons

0/4 · 0%
  1. 5Lazy Evaluation, Actions & DAGTransformations vs actions, jobs, stages and tasks.
  2. 6GroupBy & AggregationsgroupBy, agg, pivot and handling nulls.
  3. 7Functions Reference: Column & Row TransformationsThe built-in column functions every data engineer uses, grouped by purpose.
  4. 9JoinsJoin types, broadcast joins and duplicate column issues.

Advanced lessons

0/5 · 0%
  1. 8Functions Reference: Complex Types, Aggregations & WindowsArrays, maps, structs and JSON, plus aggregate and window functions for real pipelines.
  2. 10Window Functionsrow_number, rank, lag/lead and running totals.
  3. 11Spark SQL & UDFsTemp views, SQL queries and when (not) to write UDFs.
  4. 12Partitions, repartition & coalesceControlling parallelism and output file counts.
  5. 13Caching & Persistencecache, persist storage levels and when to unpersist.

Expert lessons

0/3 · 0%
  1. 14Performance Tuning & Data SkewAQE, skew handling, salting and reading the Spark UI.
  2. 15Delta Lake & MERGEACID tables, upserts, time travel and SCD Type 2.
  3. 16Structured StreamingStreams as unbounded tables, watermarks and checkpoints.

Interview preparation

Module — Core Technical Interview Q&A (Top Questions & Answers)

6 questions · 0 completed · 0%

Beginner: 1 · Freshers · 0 yearsBasic: 1 · Entry level · 0–1 yearsIntermediate: 1 · Junior · 1–3 yearsAdvanced: 1 · Mid-level · 3–5 yearsExpert: 1 · Senior · 5–8 yearsArchitect / Specialist: 1 · Specialist · 8–10+ years
  1. Beginner · Fundamentals

    Why does a DataFrame filter not immediately process the data?

    A DataFrame transformation builds a logical plan. An action such as count, show or write requests execution; creating another DataFrame does not itself materialize rows.

  2. Basic · Execution

    How do jobs, stages and tasks relate to a shuffle?

    An action normally creates a job. Spark divides its execution into stages separated by shuffle dependencies; each stage processes partitions through tasks.

  3. Intermediate · Data quality

    Why is dropDuplicates not enough to select the latest customer record?

    dropDuplicates on a key keeps a representative row, not the row with the latest timestamp. Its selected row is not a documented ordering guarantee.

  4. Advanced · Performance

    How would you diagnose a single long-running join task?

    Compare task durations, shuffle read sizes and spill across the stage. A small set of large partitions suggests key skew; uniform slow tasks point to a different bottleneck.

  5. Expert · Streaming

    Does a watermark guarantee all late data is processed exactly once?

    A watermark expresses an event-time lateness boundary used by stateful operators to finalize windows and evict state. It is not a blanket exactly-once guarantee.

  6. Architect / Specialist · Pipeline architecture

    How would you make a large incremental pipeline restartable and auditable?

    Define source offsets or versions, target keys, a deterministic conflict rule and commit boundaries. A restart must repeat or resume work without creating different business results.

Module — Coding Interview Problems & Solutions (Hands-on)

6 questions · 0 completed · 0%

Beginner: 1 · Freshers · 0 yearsBasic: 1 · Entry level · 0–1 yearsIntermediate: 1 · Junior · 1–3 yearsAdvanced: 1 · Mid-level · 3–5 yearsExpert: 1 · Senior · 5–8 yearsArchitect / Specialist: 1 · Specialist · 8–10+ years
  1. Beginner · DataFrames

    Filter positive non-null transaction amounts

    Keep rows with amount strictly greater than zero, preserving id and amount.

  2. Basic · Aggregation

    Count whitespace-separated words with DataFrame expressions

    Count case-sensitive whitespace-separated nonempty words across non-null input lines.

  3. Intermediate · Windows

    Select the highest customer version with an explicit tie-breaker

    Keep the highest version for each customer; break same-version ties by lexicographically smallest non-null city.

  4. Advanced · Joins

    Enrich orders with a small unique customer dimension

    Left-join orders to a small customer dimension without losing unmatched orders.

  5. Expert · Window frames

    Compute a rolling mean of the last three observations per entity

    Compute the mean over the current and preceding two rows per id ordered by unique day.

  6. Architect / Specialist · Clickstream processing

    Assign per-user batch sessions with a strict inactivity gap

    Create a new per-user session at the first event or when the gap from the previous event exceeds ten units.

A–Z index

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.