← Python roadmap

Build a bounded JSON-lines record parser

Architect / SpecialistSpecialist · 8–10+ yearsProduction streaming

Python 3.10+; standard library only. GIL discussion refers to conventional CPython builds, not optional free-threaded builds.

Problem statement

Yield (line_number, object) for nonblank JSON-object lines; fail with a line-specific error on oversized, invalid or non-object records.

Validate ingestion records without accumulating the whole stream.

Input: An iterable of already bounded text lines and a positive maximum character count.

Output: Lazy tuples of original line number and parsed dict.

  • Upstream must bound line reading; this parser cannot undo memory allocated before receiving a line.
  • max_chars includes line terminators.
  • Non-object JSON is rejected.

Sample 1

Input: object, blank, object
Output: line numbers 1 and 3

Blank lines do not reset original numbering.

Sample 2

Input: [1,2]
Output: ValueError: object required

Arrays violate the record contract.

Brute-force solution

Parse every line into a list before returning; memory grows with the entire input.

O(total input characters) for ordinary JSON parsing.

import json
def parse_json_lines(lines,max_chars=1024):
    if max_chars<1: raise ValueError("invalid limit")
    for number,line in enumerate(lines,1):
        if len(line)>max_chars: raise ValueError(f"line {number}: too long")
        if not line.strip():continue
        try:value=json.loads(line)
        except json.JSONDecodeError as exc:raise ValueError(f"line {number}: invalid JSON") from exc
        if not isinstance(value,dict):raise ValueError(f"line {number}: object required")
        yield number,value
print(list(parse_json_lines(['{"id":1}\n',"\n",'{"id":2}'])))
Optimized solution

Yield one validated object at a time and retain only the current record.

  1. Check line length.
  2. Skip blank input.
  3. Parse JSON and validate its type.
  4. Yield the original line number.
import json
def parse_json_lines(lines,max_chars=1024):
    if max_chars<1: raise ValueError("invalid limit")
    for number,line in enumerate(lines,1):
        if len(line)>max_chars: raise ValueError(f"line {number}: too long")
        if not line.strip():continue
        try:value=json.loads(line)
        except json.JSONDecodeError as exc:raise ValueError(f"line {number}: invalid JSON") from exc
        if not isinstance(value,dict):raise ValueError(f"line {number}: object required")
        yield number,value
print(list(parse_json_lines(['{"id":1}\n',"\n",'{"id":2}'])))
Complexity analysis

Time: O(total input characters) for ordinary JSON parsing.

Auxiliary space: O(max record size) working memory; excludes upstream buffers and consumer accumulation.

Test cases
  • Blank lines.
  • Invalid JSON.
  • Oversized record.
  • Non-object JSON.
assert list(parse_json_lines(['{"id":1}'," ",'{"id":2}']))==[(1,{"id":1}),(3,{"id":2})]
for lines in [["[1]"],["bad"],["xxxxx"]]:
    try:list(parse_json_lines(lines,4))
    except ValueError:pass
    else:raise AssertionError("invalid record accepted")
Common mistakes
  • Loading the full input before yielding.
  • Claiming downstream validation bounds upstream reads.
Key takeaways
  • Bound producer and consumer.
  • Keep original line identities.
  • Define validation failures.
Follow-up challenges
  • How would you enforce a byte limit during reading?
  • How would you quarantine invalid lines without leaking payloads?