Topic 11 of 28 · Advanced
Spark SQL & UDFs
Temp views, SQL queries and when (not) to write UDFs.
Concept
Register a DataFrame as a temp view and query it with spark.sql — same optimizer, same speed as the DataFrame API.
A Python UDF runs row by row in Python workers, which is slow and blocks optimisation. Use built-in functions first.
If you must, use a pandas_udf (vectorised via Apache Arrow), which processes batches and is much faster.
Code (PySpark 3.5)
orders.createOrReplaceTempView("orders")
spark.sql("""
SELECT city, SUM(amount) AS revenue
FROM orders GROUP BY city ORDER BY revenue DESC
""").show()
import pandas as pd
from pyspark.sql.functions import pandas_udf
@pandas_udf("double")
def with_gst(amount: pd.Series) -> pd.Series:
return amount * 1.18
orders.withColumn("gross", with_gst("amount")).show(3)Remember
- SQL and DataFrame API compile to the same plan.
- Built-in > pandas_udf > Python UDF.
- UDFs are black boxes to Catalyst.
Interview questions
Why are Python UDFs slow?
Each row is serialized from the JVM to a Python process and back, and Catalyst can't optimise inside the UDF.
Temp view vs global temp view?
A temp view lives for the SparkSession; a global temp view lives for the application and is accessed as global_temp.name.
Lesson test
Take an 8-question test on “Spark SQL & UDFs” — concepts, output prediction, bug spotting and interview questions.