← PySpark roadmap

Topic 4 of 28 · Beginner

Selecting, Filtering & Columns

select, withColumn, filter, when and built-in functions.

Concept

Transformations return a new DataFrame. select picks columns, withColumn adds/replaces one, filter/where keeps rows, drop removes columns.

Use built-in functions from pyspark.sql.functions (col, lit, when, upper, to_date). They run in the JVM and are much faster than Python UDFs.

when(...).otherwise(...) is Spark's CASE WHEN.

Code (PySpark 3.5)

from pyspark.sql import functions as F

result = (orders
    .filter(F.col("amount") > 0)
    .withColumn("city", F.upper("city"))
    .withColumn("bucket",
        F.when(F.col("amount") >= 5000, "high")
         .when(F.col("amount") >= 1000, "medium")
         .otherwise("low"))
    .withColumn("year", F.year("order_date"))
    .select("order_id", "city", "amount", "bucket", "year"))
result.show(5)

Remember

  • Always import functions as F.
  • Built-ins beat UDFs.
  • Chaining many withColumn calls bloats the plan — use select for many columns.

Interview questions

select vs withColumn?

select returns only the listed columns (and can compute many at once). withColumn adds or replaces one column, keeping the rest.

How do you rename columns?

withColumnRenamed('old', 'new'), or select(F.col('old').alias('new')).

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Selecting, Filtering & Columns” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.