Structure-Aware Tokenization for JSON

We present json-tokenizer, a structure-aware tokenizer that assigns dedicated single tokens to JSON grammar elements, learns a compact key vocabulary from training data, and applies byte-pair encoding (BPE) only to value content. General-purpose subword tokenizers such as cl100k_base treat JSON as flat text, splitting structural syntax characters across multiple tokens and re-encoding identical key names in every object. On schema-repetitive JSON workloads -- including GeoJSON features, observability telemetry, and Kubernetes manifests -- json-tokenizer achieves 5-15% token savings over cl100k_base while using a vocabulary approximately 90x smaller (1,100–4,100 tokens vs. 100,256). Batch encoding of JSON arrays yields savings up to 9.3%. All encodings are lossless: decoded output is byte-identical to the original JSON across 4,200+ test objects. Results are stable across five random seeds (standard deviation <0.01%).

We provide an open-source, zero-dependency Python implementation with HuggingFace PreTrainedTokenizer compatibility, benchmarks on five real-world dataset classes, throughput measurements, a vocabulary size sweep analysis, and an honest comparison showing where general-purpose tokenizers remain superior (prose- heavy JSON).

The tokenizer is designed for LLM structured output pipelines, API response caching, and observability log ingestion, where schema repetition dominates and token budgets directly impact cost and latency.