Topic 24 of 28 · Beginner
Count whitespace-separated words with DataFrame expressions
Count case-sensitive whitespace-separated nonempty words across non-null input lines.
Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.
Problem statement
Count case-sensitive whitespace-separated nonempty words across non-null input lines.
Summarize a small text corpus without Python row callbacks.
Input: DataFrame line string.
Output: word string, count long.
- NULL lines contribute no words.
- Whitespace separates tokens; punctuation remains part of a word.
Sample 1
Input: a b a / b c Output: a2,b2,c1
Counts combine across lines.
Sample 2
Input: blank line Output: no word rows
Empty tokens are removed.
Brute-force solution
collect all lines then count in Python, limited by driver memory.
O(total input characters + tokens), plus shuffle and displayed ordering.
from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("interview-check").config("spark.sql.shuffle.partitions","2").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
df=spark.createDataFrame([("a b a",),("b c",)],"line string")
words=df.select(F.explode(F.split(F.col("line"),r"\s+")).alias("word")).filter("word != ''")
result=words.groupBy("word").count().orderBy("word")
rows=[tuple(r) for r in result.collect()]
assert rows==[("a",2),("b",2),("c",1)]
print(rows)
spark.stop()
Optimized solution
Explode split tokens and groupBy word; Spark distributes the count aggregation.
- Split on whitespace.
- Explode token arrays.
- Remove empty tokens.
- Group and count.
from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("interview-check").config("spark.sql.shuffle.partitions","2").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
df=spark.createDataFrame([("a b a",),("b c",)],"line string")
words=df.select(F.explode(F.split(F.col("line"),r"\s+")).alias("word")).filter("word != ''")
result=words.groupBy("word").count().orderBy("word")
rows=[tuple(r) for r in result.collect()]
assert rows==[("a",2),("b",2),("c",1)]
print(rows)
spark.stop()
Complexity analysis
Time: O(total input characters + tokens), plus shuffle and displayed ordering.
Auxiliary space: O(distinct word state) distributed.
Test cases
- Blank.
- Repeated whitespace.
- NULL line.
Supplied script asserts a2, b2, c1.Common mistakes
- Counting empty split tokens.
- Calling collect before distributed aggregation.
Key takeaways
- Define tokenization.
- Use distributed aggregation.
- Never collect large raw input.
Follow-up challenges
- When is reduceByKey better than groupByKey?
- How would Unicode tokenization change the contract?