← PySpark roadmap

Topic 11 of 28 · Advanced

Spark SQL & UDFs

Temp views, SQL queries and when (not) to write UDFs.

Concept

Register a DataFrame as a temp view and query it with spark.sql — same optimizer, same speed as the DataFrame API.

A Python UDF runs row by row in Python workers, which is slow and blocks optimisation. Use built-in functions first.

If you must, use a pandas_udf (vectorised via Apache Arrow), which processes batches and is much faster.

Code (PySpark 3.5)

orders.createOrReplaceTempView("orders")
spark.sql("""
  SELECT city, SUM(amount) AS revenue
  FROM orders GROUP BY city ORDER BY revenue DESC
""").show()

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def with_gst(amount: pd.Series) -> pd.Series:
    return amount * 1.18

orders.withColumn("gross", with_gst("amount")).show(3)

Remember

  • SQL and DataFrame API compile to the same plan.
  • Built-in > pandas_udf > Python UDF.
  • UDFs are black boxes to Catalyst.

Interview questions

Why are Python UDFs slow?

Each row is serialized from the JVM to a Python process and back, and Catalyst can't optimise inside the UDF.

Temp view vs global temp view?

A temp view lives for the SparkSession; a global temp view lives for the application and is accessed as global_temp.name.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Spark SQL & UDFs” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.