← PySpark roadmap

Topic 21 of 28 · Expert

Does a watermark guarantee all late data is processed exactly once?

A watermark expresses an event-time lateness boundary used by stateful operators to finalize windows and evict state. It is not a blanket exactly-once guarantee.

ExpertSenior · 5–8 yearsStreaming

Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.

Detailed answer

A watermark expresses an event-time lateness boundary used by stateful operators to finalize windows and evict state. It is not a blanket exactly-once guarantee.

Data within the supported lateness threshold is protected by the operator contract; sufficiently late data may be dropped. The watermark advances with observed event time, not wall-clock time alone.

Exactly-once outcomes depend on replayable sources, checkpoints and transactional or idempotent sinks. foreachBatch is at-least-once unless the batch write handles replays safely.

Internal working

State stores hold partial aggregates and deduplication keys. Operator type, output mode, multiple-input watermark policy and event-time progress determine cleanup behavior.

Practical scenario

A consumer restarts and replays a batch into an append-only external sink. Duplicate rows appear despite a configured watermark.

from pyspark.sql import SparkSession
spark=SparkSession.builder.master("local[2]").appName("stream").getOrCreate()
# A bounded rate source illustrates a streaming plan; production needs a durable checkpoint.
df=spark.readStream.format("rate").option("rowsPerSecond",1).load()
print(df.isStreaming)
spark.stop()
Common mistakes
  • Treating watermark as wall-clock expiry.
  • Assuming foreachBatch writes exactly once automatically.
Key takeaways
  • Define acceptable lateness.
  • Bound operator state.
  • Make sinks replay safe.
Follow-up challenges
  • How would you quarantine late arrivals?
  • How do checkpoint loss and sink identity interact?
Practise PySpark with the AI mentor

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.