Technical Note #1: Design Trade-offs and Failure Modes in a Real-Time Market Data Pipeline

Technical Note #1 in a series on a distributed real-time market-data system.

This note studies the design and operation of a live market-data pipeline that ingests exchange updates, reconstructs per-symbol Level 2 order books, and computes derived streams online. The focus is on how the system remains operational under irregular input, changing load, saturation, recovery events, and bounded compute resources.

The analysis covers the pipeline design, the deployed implementation, and a live observation window around the U.S. market open. The case study covers 934 order books and more than 280 million output records, and shows how backlog, delay, explicit loss, and snapshot recovery appear as the system approaches saturation.