Topic 16 of 28 · Expert
Structured Streaming
Streams as unbounded tables, watermarks and checkpoints.
Concept
Structured Streaming treats a stream as an ever-growing table; you write the same DataFrame code and Spark runs it incrementally in micro-batches.
Watermarks tell Spark how late data may arrive so it can finalise windows and drop old state.
Checkpoints store progress and state so a restarted query resumes exactly where it stopped; with idempotent sinks this gives exactly-once results.
Code (PySpark 3.5)
events = (spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "clicks").load()
.selectExpr("CAST(value AS STRING) json")
.select(F.from_json("json", "user STRING, ts TIMESTAMP").alias("e"))
.select("e.*"))
counts = (events.withWatermark("ts", "10 minutes")
.groupBy(F.window("ts", "5 minutes"), "user").count())
(counts.writeStream.outputMode("append")
.format("delta")
.option("checkpointLocation", "/chk/clicks")
.start("/lake/click_counts"))Remember
- Always set a checkpointLocation.
- Watermarks bound state size.
- Output modes: append, update, complete.
Interview questions
What is a watermark?
A threshold of event-time lateness; data older than max_event_time − watermark is dropped, letting Spark finalise windows and clean state.
How does Spark achieve exactly-once?
Replayable sources + checkpointed offsets/state + idempotent or transactional sinks.
Lesson test
Take an 8-question test on “Structured Streaming” — concepts, output prediction, bug spotting and interview questions.