2 · DataFrames · lesson 5 of 20

Creating DataFrames

From lists, dicts, pandas, CSV, JSON, Parquet — every common source.

Python
from pyspark.sql import Row

# From a Python list of tuples with an explicit schema
df1 = spark.createDataFrame(
    [(1, "Ada"), (2, "Grace"), (3, "Linus")],
    schema="id INT, name STRING",
)

# From a list of Row objects
df2 = spark.createDataFrame([Row(id=1, name="Ada")])

# From files
df_csv     = spark.read.option("header", True).csv("/data/*.csv")
df_json    = spark.read.json("/data/events.json")
df_parquet = spark.read.parquet("/data/orders")
TIP
Prefer Parquet for storage — it's columnar, compressed, and preserves schema, so Spark can read only the columns it needs.
Loading 3D scene…
Key takeaways
  • ✓createDataFrame turns Python data into a distributed DataFrame.
  • ✓spark.read.<format> covers CSV, JSON, Parquet, ORC, Avro, JDBC, Delta…
  • ✓Parquet is the default choice for analytical workloads.