← PySpark roadmap

Topic 23 of 28 · Beginner

Filter positive non-null transaction amounts

Keep rows with amount strictly greater than zero, preserving id and amount.

BeginnerFreshers · 0 yearsDataFrames

Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.

Problem statement

Keep rows with amount strictly greater than zero, preserving id and amount.

Remove invalid amounts from a small payment input.

Input: DataFrame schema id int, amount int nullable.

Output: Filtered DataFrame.

  • IDs unique.
  • NULL and zero do not qualify.
  • collect used only for tiny sample verification.

Sample 1

Input: (1,10),(2,-1),(3,NULL)
Output: (1,10)

NULL comparison is unknown and filtered out.

Sample 2

Input: (1,0)
Output: empty

Zero is not positive.

Brute-force solution

A Python row UDF could test each amount but adds serialization overhead and blocks predicate optimization.

O(n) scan; ordered demonstration adds sorting.

from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("interview-check").config("spark.sql.shuffle.partitions","2").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
df=spark.createDataFrame([(1,10),(2,-1),(3,None)],"id int, amount int")
result=df.filter(F.col("amount")>0).orderBy("id")
rows=[tuple(r) for r in result.collect()]
assert rows==[(1,10)]
print(rows)
spark.stop()
Optimized solution

Use a built-in column predicate so Catalyst can optimize the filter.

  1. Create explicit schema.
  2. Filter amount>0.
  3. Order only for small demonstration output.
from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("interview-check").config("spark.sql.shuffle.partitions","2").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
df=spark.createDataFrame([(1,10),(2,-1),(3,None)],"id int, amount int")
result=df.filter(F.col("amount")>0).orderBy("id")
rows=[tuple(r) for r in result.collect()]
assert rows==[(1,10)]
print(rows)
spark.stop()
Complexity analysis

Time: O(n) scan; ordered demonstration adds sorting.

Auxiliary space: Distributed partition workspace; O(sample size) only for sample collect.

Test cases
  • Null.
  • Zero.
  • Empty frame.
The supplied script asserts exactly [(1,10)].
Common mistakes
  • Using a Python UDF for a built-in comparison.
  • Collecting production output on the driver.
Key takeaways
  • Use built-in expressions.
  • State NULL semantics.
  • Keep driver checks bounded.
Follow-up challenges
  • How does predicate pushdown depend on source format?
  • How would NULL rows be quarantined?
Practise PySpark with the AI mentor

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.