← PySpark roadmap

Topic 19 of 28 · Intermediate

Why is dropDuplicates not enough to select the latest customer record?

dropDuplicates on a key keeps a representative row, not the row with the latest timestamp. Its selected row is not a documented ordering guarantee.

IntermediateJunior · 1–3 yearsData quality

Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.

Detailed answer

dropDuplicates on a key keeps a representative row, not the row with the latest timestamp. Its selected row is not a documented ordering guarantee.

Use row_number over a partition ordered by descending event timestamp and a stable secondary key. Define how to handle null timestamps and exact ties.

A deterministic tie-breaker is necessary for repeatable pipeline results. If two rows conflict at the same version, a data-quality failure may be preferable to arbitrarily choosing one.

Internal working

Deduplication aggregates by key; a window ordered inside each key expresses version precedence and keeps the chosen row's other fields together.

Practical scenario

Two customer updates arrive out of order. Dropping duplicates can restore an older address after rerun.

from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("latest").getOrCreate()
df=spark.createDataFrame([(1,1,"A"),(1,2,"B")],"id int, version int, city string")
w=Window.partitionBy("id").orderBy(F.desc("version"))
df.withColumn("rn",F.row_number().over(w)).filter("rn=1").show()
spark.stop()
Common mistakes
  • Assuming global sorting controls dropDuplicates.
  • Omitting tie-breaking rules.
Key takeaways
  • Representative is not latest.
  • Use deterministic version order.
  • Validate conflicting versions.
Follow-up challenges
  • What differs for streaming deduplication?
  • How should null event times be handled?
Practise PySpark with the AI mentor

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.