Topic 4 of 28 · Beginner
Selecting, Filtering & Columns
select, withColumn, filter, when and built-in functions.
Concept
Transformations return a new DataFrame. select picks columns, withColumn adds/replaces one, filter/where keeps rows, drop removes columns.
Use built-in functions from pyspark.sql.functions (col, lit, when, upper, to_date). They run in the JVM and are much faster than Python UDFs.
when(...).otherwise(...) is Spark's CASE WHEN.
Code (PySpark 3.5)
from pyspark.sql import functions as F
result = (orders
.filter(F.col("amount") > 0)
.withColumn("city", F.upper("city"))
.withColumn("bucket",
F.when(F.col("amount") >= 5000, "high")
.when(F.col("amount") >= 1000, "medium")
.otherwise("low"))
.withColumn("year", F.year("order_date"))
.select("order_id", "city", "amount", "bucket", "year"))
result.show(5)Remember
- Always import functions as F.
- Built-ins beat UDFs.
- Chaining many withColumn calls bloats the plan — use select for many columns.
Interview questions
select vs withColumn?
select returns only the listed columns (and can compute many at once). withColumn adds or replaces one column, keeping the rest.
How do you rename columns?
withColumnRenamed('old', 'new'), or select(F.col('old').alias('new')).
Lesson test
Take an 8-question test on “Selecting, Filtering & Columns” — concepts, output prediction, bug spotting and interview questions.