PIZD: Adaptive Column Models and Structural Factorization for Lossless Compression of Logs and Structured Data

Logs and structured records contain redundancy within fields and across related fields that can be exposed before general-purpose compression. We present PIZD, a lossless compression system that separates recognized records into columns, selects data-dependent column representations, and reconstructs the original bytes, including layout and record order. Its models combine numeric deltas, categorical dictionaries, shared-prefix coding, inter-column prediction, structural template factorization (PZTF03), deterministic 4-state Markov literal character contexts, and dynamic keyword pre-tokenization for structured JSON/NDJSON records. Native LZ/Huffman coding, optional four-state rANS channels, and a Hybrid Zstandard backend provide alternative coding paths; candidate selection accounts for complete representation metadata and reconstruction costs.

On an exploratory panel of 18 generated enterprise workloads totaling 5.67 MB, PIZD 6.0.0-rc8 Native small achieves an aggregate compressed size of 0.25 MB (4.38% compression ratio, 95.62% space saving), winning size comparisons on 17 of 18 inputs against the best measured reference. Hybrid fast compresses the panel in 33.8 ms (167.8 MB/s) with verified decompression at 781.1 MB/s. On the 30-input public-excerpt panel, version 6.0.0-rc8 introduces Dynamic Keyword Pre-Tokenization and 4-state Markov character contexts. In aggregate across the 30 public benchmarks (60 restorations), Native thin drops from 757,504 B to 713,735 B (-43,769 B net reduction, -5.78%), while Native thin with template factorization drops from 731,403 B to 687,537 B (-43,866 B net reduction, -6.00%), saving 22,163 net bytes across adverse GitHub JSONL files with zero stored-size regressions. Every restored archive achieves 100% bit-for-bit SHA-256 lossless identity. The evidence supports structural modeling as a useful size–time tradeoff, while broader real-workload validation, specialized log-compressor comparisons and independent replication remain open.