Topic 21 of 28 · Expert
Does a watermark guarantee all late data is processed exactly once?
A watermark expresses an event-time lateness boundary used by stateful operators to finalize windows and evict state. It is not a blanket exactly-once guarantee.
Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.
Detailed answer
A watermark expresses an event-time lateness boundary used by stateful operators to finalize windows and evict state. It is not a blanket exactly-once guarantee.
Data within the supported lateness threshold is protected by the operator contract; sufficiently late data may be dropped. The watermark advances with observed event time, not wall-clock time alone.
Exactly-once outcomes depend on replayable sources, checkpoints and transactional or idempotent sinks. foreachBatch is at-least-once unless the batch write handles replays safely.
Internal working
State stores hold partial aggregates and deduplication keys. Operator type, output mode, multiple-input watermark policy and event-time progress determine cleanup behavior.
Practical scenario
A consumer restarts and replays a batch into an append-only external sink. Duplicate rows appear despite a configured watermark.
from pyspark.sql import SparkSession
spark=SparkSession.builder.master("local[2]").appName("stream").getOrCreate()
# A bounded rate source illustrates a streaming plan; production needs a durable checkpoint.
df=spark.readStream.format("rate").option("rowsPerSecond",1).load()
print(df.isStreaming)
spark.stop()Common mistakes
- Treating watermark as wall-clock expiry.
- Assuming foreachBatch writes exactly once automatically.
Key takeaways
- Define acceptable lateness.
- Bound operator state.
- Make sinks replay safe.
Follow-up challenges
- How would you quarantine late arrivals?
- How do checkpoint loss and sink identity interact?