← PySpark roadmap

Topic 16 of 28 · Expert

Structured Streaming

Streams as unbounded tables, watermarks and checkpoints.

Concept

Structured Streaming treats a stream as an ever-growing table; you write the same DataFrame code and Spark runs it incrementally in micro-batches.

Watermarks tell Spark how late data may arrive so it can finalise windows and drop old state.

Checkpoints store progress and state so a restarted query resumes exactly where it stopped; with idempotent sinks this gives exactly-once results.

Code (PySpark 3.5)

events = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "clicks").load()
    .selectExpr("CAST(value AS STRING) json")
    .select(F.from_json("json", "user STRING, ts TIMESTAMP").alias("e"))
    .select("e.*"))

counts = (events.withWatermark("ts", "10 minutes")
    .groupBy(F.window("ts", "5 minutes"), "user").count())

(counts.writeStream.outputMode("append")
    .format("delta")
    .option("checkpointLocation", "/chk/clicks")
    .start("/lake/click_counts"))

Remember

  • Always set a checkpointLocation.
  • Watermarks bound state size.
  • Output modes: append, update, complete.

Interview questions

What is a watermark?

A threshold of event-time lateness; data older than max_event_time − watermark is dropped, letting Spark finalise windows and clean state.

How does Spark achieve exactly-once?

Replayable sources + checkpointed offsets/state + idempotent or transactional sinks.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Structured Streaming” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.