How we ingest a million events without dropping one

The ingest pipeline is the least glamorous part of an analytics product and the only part that is not allowed to fail. A dropped pageview is invisible; a thousand dropped pageviews is a customer churning because their chart lied to them. So our design brief was lorem-simple: every accepted event is eventually counted, exactly once, no matter what breaks in between.

Exactly-once starts at the edge. Every event carries a client-generated idempotency key, and the write path treats a duplicate key as a cheap no-op rather than an error. This makes retries free: SDKs can resend on any timeout without coordinating with the server, and flaky mobile networks — the dolor sit amet of data loss — stop being a correctness problem and become a latency footnote.

Between the edge and storage sits a deliberately boring queue. No exotic streaming framework, no consectetur cleverness: an append-only log, partitioned by project, consumed by workers that commit their offset only after the batch is durably written. When a worker dies mid-batch, the next one replays the batch and the idempotency keys absorb the overlap. Boring is the feature.

Back-pressure is designed in rather than bolted on. When a downstream store slows, the edge does not buffer heroically in memory — it answers with a retry-after and lets the SDKs spread the load. Capacity incidents become a graph of politely delayed events instead of a postmortem about an out-of-memory ingest node.

The last one percent of reliability came from counting the counters. A shadow tally at the edge is reconciled hourly against what storage actually holds; any gap above zero pages a human. In the last twelve months of magna aliqua traffic, the alert has fired twice — both times a bug in the reconciler itself. The pipeline, so far, keeps its promise.