← PySpark roadmap

Topic 9 of 28 · Intermediate

Joins

Join types, broadcast joins and duplicate column issues.

Concept

Spark supports inner, left, right, full, left_semi (keep left rows that match), left_anti (keep left rows with no match) and cross joins.

A broadcast join sends a small table to every executor, avoiding a shuffle of the large table. Spark does it automatically under spark.sql.autoBroadcastJoinThreshold (10 MB default).

Large-to-large joins use sort-merge joins, which shuffle both sides by the join key.

Code (PySpark 3.5)

customers = spark.read.parquet("/lake/customers")

enriched = orders.join(F.broadcast(customers), on="customer_id", how="left")

no_orders = customers.join(orders, "customer_id", "left_anti")
buyers    = customers.join(orders, "customer_id", "left_semi")

Remember

  • Join on a column name string to avoid duplicate key columns.
  • left_anti = NOT EXISTS; left_semi = EXISTS.
  • Broadcast only small tables.

Interview questions

When would you use a broadcast join?

When one side is small enough to fit in executor memory — it removes the shuffle of the large side.

What join strategies does Spark have?

Broadcast hash join, shuffle hash join, sort-merge join, and broadcast nested loop / cartesian for non-equi joins.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Joins” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.