2 · DataFrames · lesson 7 of 20

select, filter & withColumn

The three verbs you'll use in every single script.

Python

from pyspark.sql import functions as F

(df
    .select("id", "name", (F.col("amount") * 1.2).alias("gross"))
    .filter((F.col("country") == "FR") & (F.col("amount") > 100))
    .withColumn("year", F.year("created_at"))
    .withColumnRenamed("name", "customer")
    .drop("id")
    .show()
)

▸select — pick / compute a subset of columns.
▸filter (alias: where) — keep matching rows.
▸withColumn — add or replace a single column.
▸F.col('x') vs df.x — same thing; F.col works with any DataFrame.

TIP

Chain everything through pyspark.sql.functions (imported as F). Never use Python if/else on Column objects — use F.when(...).otherwise(...) instead.

Loading 3D scene…

Key takeaways

✓select / filter / withColumn cover the vast majority of transforms.
✓Column expressions are lazy — no data moves until an action fires.
✓Use F.when / F.coalesce / F.lit for conditional and constant columns.