Topic 23 of 28 · Beginner
Filter positive non-null transaction amounts
Keep rows with amount strictly greater than zero, preserving id and amount.
Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.
Problem statement
Keep rows with amount strictly greater than zero, preserving id and amount.
Remove invalid amounts from a small payment input.
Input: DataFrame schema id int, amount int nullable.
Output: Filtered DataFrame.
- IDs unique.
- NULL and zero do not qualify.
- collect used only for tiny sample verification.
Sample 1
Input: (1,10),(2,-1),(3,NULL) Output: (1,10)
NULL comparison is unknown and filtered out.
Sample 2
Input: (1,0) Output: empty
Zero is not positive.
Brute-force solution
A Python row UDF could test each amount but adds serialization overhead and blocks predicate optimization.
O(n) scan; ordered demonstration adds sorting.
from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("interview-check").config("spark.sql.shuffle.partitions","2").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
df=spark.createDataFrame([(1,10),(2,-1),(3,None)],"id int, amount int")
result=df.filter(F.col("amount")>0).orderBy("id")
rows=[tuple(r) for r in result.collect()]
assert rows==[(1,10)]
print(rows)
spark.stop()
Optimized solution
Use a built-in column predicate so Catalyst can optimize the filter.
- Create explicit schema.
- Filter amount>0.
- Order only for small demonstration output.
from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("interview-check").config("spark.sql.shuffle.partitions","2").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
df=spark.createDataFrame([(1,10),(2,-1),(3,None)],"id int, amount int")
result=df.filter(F.col("amount")>0).orderBy("id")
rows=[tuple(r) for r in result.collect()]
assert rows==[(1,10)]
print(rows)
spark.stop()
Complexity analysis
Time: O(n) scan; ordered demonstration adds sorting.
Auxiliary space: Distributed partition workspace; O(sample size) only for sample collect.
Test cases
- Null.
- Zero.
- Empty frame.
The supplied script asserts exactly [(1,10)].Common mistakes
- Using a Python UDF for a built-in comparison.
- Collecting production output on the driver.
Key takeaways
- Use built-in expressions.
- State NULL semantics.
- Keep driver checks bounded.
Follow-up challenges
- How does predicate pushdown depend on source format?
- How would NULL rows be quarantined?