2 · DataFrames · lesson 10 of 20

Spark SQL

Every DataFrame op is also an SQL query — and you can mix both freely.

Python

df.createOrReplaceTempView("orders")

top = spark.sql("""
    SELECT country, SUM(amount) AS revenue
    FROM orders
    WHERE created_at >= '2024-01-01'
    GROUP BY country
    ORDER BY revenue DESC
    LIMIT 10
""")

top.show()

NOTE

spark.sql returns a DataFrame. You can .filter() or .join() it further exactly like any other DataFrame — SQL and the DataFrame API are two views of the same plan.

Key takeaways

✓createOrReplaceTempView exposes a DataFrame to SQL by name.
✓spark.sql(...) → DataFrame; the two APIs interoperate freely.
✓Use SQL for exploration, the DataFrame API for reusable pipelines.