← PySpark roadmap

Topic 3 of 28 · Beginner

Reading & Writing Data

CSV, JSON, Parquet, schemas and save modes.

Concept

spark.read loads data from CSV, JSON, Parquet, ORC, JDBC and more. Parquet is columnar and compressed — the default choice for analytics.

inferSchema scans the data and is slow; in production, define an explicit schema with StructType.

df.write saves data with a mode: overwrite, append, ignore or error. partitionBy writes folders per value for faster filtering later.

Code (PySpark 3.5)

from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DateType

schema = StructType([
    StructField("order_id", IntegerType(), False),
    StructField("city", StringType(), True),
    StructField("amount", IntegerType(), True),
    StructField("order_date", DateType(), True),
])

orders = spark.read.csv("/data/orders.csv", header=True, schema=schema)

(orders.write
   .mode("overwrite")
   .partitionBy("order_date")
   .parquet("/lake/orders"))

Remember

  • Prefer Parquet over CSV.
  • Explicit schema > inferSchema in production.
  • Don't partitionBy a high-cardinality column.

Interview questions

Why is Parquet preferred?

It's columnar (reads only needed columns), compressed, stores the schema and supports predicate pushdown.

How do you handle corrupt records?

Use mode PERMISSIVE with columnNameOfCorruptRecord, DROPMALFORMED, or FAILFAST depending on how strict the pipeline must be.

Practise PySpark with the AI mentor

Lesson test

Take an 8-question test on “Reading & Writing Data” — concepts, output prediction, bug spotting and interview questions.

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.