Build a bounded JSON-lines record parser
Python 3.10+; standard library only. GIL discussion refers to conventional CPython builds, not optional free-threaded builds.
Problem statement
Yield (line_number, object) for nonblank JSON-object lines; fail with a line-specific error on oversized, invalid or non-object records.
Validate ingestion records without accumulating the whole stream.
Input: An iterable of already bounded text lines and a positive maximum character count.
Output: Lazy tuples of original line number and parsed dict.
- Upstream must bound line reading; this parser cannot undo memory allocated before receiving a line.
- max_chars includes line terminators.
- Non-object JSON is rejected.
Sample 1
Input: object, blank, object Output: line numbers 1 and 3
Blank lines do not reset original numbering.
Sample 2
Input: [1,2] Output: ValueError: object required
Arrays violate the record contract.
Brute-force solution
Parse every line into a list before returning; memory grows with the entire input.
O(total input characters) for ordinary JSON parsing.
import json
def parse_json_lines(lines,max_chars=1024):
if max_chars<1: raise ValueError("invalid limit")
for number,line in enumerate(lines,1):
if len(line)>max_chars: raise ValueError(f"line {number}: too long")
if not line.strip():continue
try:value=json.loads(line)
except json.JSONDecodeError as exc:raise ValueError(f"line {number}: invalid JSON") from exc
if not isinstance(value,dict):raise ValueError(f"line {number}: object required")
yield number,value
print(list(parse_json_lines(['{"id":1}\n',"\n",'{"id":2}'])))
Optimized solution
Yield one validated object at a time and retain only the current record.
- Check line length.
- Skip blank input.
- Parse JSON and validate its type.
- Yield the original line number.
import json
def parse_json_lines(lines,max_chars=1024):
if max_chars<1: raise ValueError("invalid limit")
for number,line in enumerate(lines,1):
if len(line)>max_chars: raise ValueError(f"line {number}: too long")
if not line.strip():continue
try:value=json.loads(line)
except json.JSONDecodeError as exc:raise ValueError(f"line {number}: invalid JSON") from exc
if not isinstance(value,dict):raise ValueError(f"line {number}: object required")
yield number,value
print(list(parse_json_lines(['{"id":1}\n',"\n",'{"id":2}'])))
Complexity analysis
Time: O(total input characters) for ordinary JSON parsing.
Auxiliary space: O(max record size) working memory; excludes upstream buffers and consumer accumulation.
Test cases
- Blank lines.
- Invalid JSON.
- Oversized record.
- Non-object JSON.
assert list(parse_json_lines(['{"id":1}'," ",'{"id":2}']))==[(1,{"id":1}),(3,{"id":2})]
for lines in [["[1]"],["bad"],["xxxxx"]]:
try:list(parse_json_lines(lines,4))
except ValueError:pass
else:raise AssertionError("invalid record accepted")Common mistakes
- Loading the full input before yielding.
- Claiming downstream validation bounds upstream reads.
Key takeaways
- Bound producer and consumer.
- Keep original line identities.
- Define validation failures.
Follow-up challenges
- How would you enforce a byte limit during reading?
- How would you quarantine invalid lines without leaking payloads?