Real-time Market Data Pipeline That Doesn't Break

Market data is the foundation everything else stands on. If ingestion hiccups, every downstream system — signals, risk, execution — inherits the damage. This is the pipeline architecture that has survived exchange outages, broker disconnects, and expiry-day tick storms.

The shape of the pipeline

Four stages, each with a single responsibility, connected by Redis:

WebSocket feeds ─▶ Normalizer ─▶ Redis streams ─▶ Consumers
 (per broker)      (one schema)    (buffer+fanout)   (strategies, storage, risk)

Broker adapters do nothing but maintain connections and stamp receive-time on raw messages. The normalizer converts every vendor's quirks into one internal schema. Redis streams give me buffering, replay, and fan-out to any number of consumers without the producers knowing they exist.

Reconnection is the main feature

Every feed will disconnect — daily, and always at the worst moment. The adapter treats reconnection as its core loop: exponential backoff with jitter, resubscription from a checkpoint, and a gap-fill request against the REST API for anything missed while down. A feed that has been up for six hours is not "stable"; it's six hours closer to its next disconnect.

Normalize once, at the edge

Every broker encodes symbols, timestamps, and depth differently. Letting that leak past the normalizer means every strategy carries vendor-specific branches forever. One internal tick schema — instrument id, exchange timestamp, receive timestamp, price, quantity, depth — and everything downstream is vendor-agnostic. Adding a new broker is a new adapter, not a platform change.

Monitoring that catches silent failures

The dangerous failure mode isn't a crash — it's a feed that stays connected and quietly stops sending. Three checks catch it:

• Heartbeat lag — alert when any instrument's last tick age exceeds its normal inter-tick gap.
• Cross-feed drift — compare tick counts across redundant feeds for the same instrument.
• Clock skew — exchange timestamp vs. receive timestamp, tracked as a distribution.

Storage is a consumer, not a stage

Persistence hangs off the same Redis stream as everything else: a writer batches ticks into columnar files for research, and a separate compactor builds candles. If the writer dies, trading is unaffected; when it recovers, it resumes from its stream offset. Decoupling storage from the hot path is the single decision that has prevented the most incidents.