Topic 3 of 28 · Beginner
Reading & Writing Data
CSV, JSON, Parquet, schemas and save modes.
Concept
spark.read loads data from CSV, JSON, Parquet, ORC, JDBC and more. Parquet is columnar and compressed — the default choice for analytics.
inferSchema scans the data and is slow; in production, define an explicit schema with StructType.
df.write saves data with a mode: overwrite, append, ignore or error. partitionBy writes folders per value for faster filtering later.
Code (PySpark 3.5)
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DateType
schema = StructType([
StructField("order_id", IntegerType(), False),
StructField("city", StringType(), True),
StructField("amount", IntegerType(), True),
StructField("order_date", DateType(), True),
])
orders = spark.read.csv("/data/orders.csv", header=True, schema=schema)
(orders.write
.mode("overwrite")
.partitionBy("order_date")
.parquet("/lake/orders"))Remember
- Prefer Parquet over CSV.
- Explicit schema > inferSchema in production.
- Don't partitionBy a high-cardinality column.
Interview questions
Why is Parquet preferred?
It's columnar (reads only needed columns), compressed, stores the schema and supports predicate pushdown.
How do you handle corrupt records?
Use mode PERMISSIVE with columnNameOfCorruptRecord, DROPMALFORMED, or FAILFAST depending on how strict the pipeline must be.
Lesson test
Take an 8-question test on “Reading & Writing Data” — concepts, output prediction, bug spotting and interview questions.