← PySpark roadmap

Topic 24 of 28 · Beginner

Count whitespace-separated words with DataFrame expressions

Count case-sensitive whitespace-separated nonempty words across non-null input lines.

BasicEntry level · 0–1 yearsAggregation

Apache Spark / PySpark 3.5+ with a local JVM. Not executable inside a plain Python browser runtime.

Problem statement

Count case-sensitive whitespace-separated nonempty words across non-null input lines.

Summarize a small text corpus without Python row callbacks.

Input: DataFrame line string.

Output: word string, count long.

  • NULL lines contribute no words.
  • Whitespace separates tokens; punctuation remains part of a word.

Sample 1

Input: a b a / b c
Output: a2,b2,c1

Counts combine across lines.

Sample 2

Input: blank line
Output: no word rows

Empty tokens are removed.

Brute-force solution

collect all lines then count in Python, limited by driver memory.

O(total input characters + tokens), plus shuffle and displayed ordering.

from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("interview-check").config("spark.sql.shuffle.partitions","2").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
df=spark.createDataFrame([("a b a",),("b c",)],"line string")
words=df.select(F.explode(F.split(F.col("line"),r"\s+")).alias("word")).filter("word != ''")
result=words.groupBy("word").count().orderBy("word")
rows=[tuple(r) for r in result.collect()]
assert rows==[("a",2),("b",2),("c",1)]
print(rows)
spark.stop()
Optimized solution

Explode split tokens and groupBy word; Spark distributes the count aggregation.

  1. Split on whitespace.
  2. Explode token arrays.
  3. Remove empty tokens.
  4. Group and count.
from pyspark.sql import SparkSession,functions as F,Window
spark=SparkSession.builder.master("local[2]").appName("interview-check").config("spark.sql.shuffle.partitions","2").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
df=spark.createDataFrame([("a b a",),("b c",)],"line string")
words=df.select(F.explode(F.split(F.col("line"),r"\s+")).alias("word")).filter("word != ''")
result=words.groupBy("word").count().orderBy("word")
rows=[tuple(r) for r in result.collect()]
assert rows==[("a",2),("b",2),("c",1)]
print(rows)
spark.stop()
Complexity analysis

Time: O(total input characters + tokens), plus shuffle and displayed ordering.

Auxiliary space: O(distinct word state) distributed.

Test cases
  • Blank.
  • Repeated whitespace.
  • NULL line.
Supplied script asserts a2, b2, c1.
Common mistakes
  • Counting empty split tokens.
  • Calling collect before distributed aggregation.
Key takeaways
  • Define tokenization.
  • Use distributed aggregation.
  • Never collect large raw input.
Follow-up challenges
  • When is reduceByKey better than groupByKey?
  • How would Unicode tokenization change the contract?
Practise PySpark with the AI mentor

Can't find a PySpark topic you need for exams or placements? Request it and we'll add it.