Topic 9 of 28 · Intermediate
Joins
Join types, broadcast joins and duplicate column issues.
Concept
Spark supports inner, left, right, full, left_semi (keep left rows that match), left_anti (keep left rows with no match) and cross joins.
A broadcast join sends a small table to every executor, avoiding a shuffle of the large table. Spark does it automatically under spark.sql.autoBroadcastJoinThreshold (10 MB default).
Large-to-large joins use sort-merge joins, which shuffle both sides by the join key.
Code (PySpark 3.5)
customers = spark.read.parquet("/lake/customers")
enriched = orders.join(F.broadcast(customers), on="customer_id", how="left")
no_orders = customers.join(orders, "customer_id", "left_anti")
buyers = customers.join(orders, "customer_id", "left_semi")Remember
- Join on a column name string to avoid duplicate key columns.
- left_anti = NOT EXISTS; left_semi = EXISTS.
- Broadcast only small tables.
Interview questions
When would you use a broadcast join?
When one side is small enough to fit in executor memory — it removes the shuffle of the large side.
What join strategies does Spark have?
Broadcast hash join, shuffle hash join, sort-merge join, and broadcast nested loop / cartesian for non-equi joins.
Lesson test
Take an 8-question test on “Joins” — concepts, output prediction, bug spotting and interview questions.