Topic 19 of 28 · Intermediate
Why is dropDuplicates not enough to select the latest customer record?
dropDuplicates on a key keeps a representative row, not the row with the latest timestamp. Its selected row is not a documented ordering guarantee.
Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.
Detailed answer
dropDuplicates on a key keeps a representative row, not the row with the latest timestamp. Its selected row is not a documented ordering guarantee.
Use row_number over a partition ordered by descending event timestamp and a stable secondary key. Define how to handle null timestamps and exact ties.
A deterministic tie-breaker is necessary for repeatable pipeline results. If two rows conflict at the same version, a data-quality failure may be preferable to arbitrarily choosing one.
Internal working
Deduplication aggregates by key; a window ordered inside each key expresses version precedence and keeps the chosen row's other fields together.
Practical scenario
Two customer updates arrive out of order. Dropping duplicates can restore an older address after rerun.
from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("latest").getOrCreate()
df=spark.createDataFrame([(1,1,"A"),(1,2,"B")],"id int, version int, city string")
w=Window.partitionBy("id").orderBy(F.desc("version"))
df.withColumn("rn",F.row_number().over(w)).filter("rn=1").show()
spark.stop()Common mistakes
- Assuming global sorting controls dropDuplicates.
- Omitting tie-breaking rules.
Key takeaways
- Representative is not latest.
- Use deterministic version order.
- Validate conflicting versions.
Follow-up challenges
- What differs for streaming deduplication?
- How should null event times be handled?