systemsadvanced 25m read

Designing for Scale · Metering and Billing

Why billing systems cannot drop a single event, the difference between at-least-once and exactly-once processing, and how to build idempotent pipelines that survive crashes without double-charging users.

The problem this post solves

A standard application API can afford to fail occasionally. If a user tries to post a comment and the server returns a 500 error, the user clicks retry. If the comment accidentally posts twice, it is an annoyance.

If a cloud provider's API usage meter drops an event, the company loses money. If it accidentally counts the event twice, the customer is overcharged, and trust is destroyed. Billing systems operate under a mandate of absolute correctness at massive scale.

This post covers how to design a high-throughput ingestion pipeline that never loses data and never double-counts it, focusing on idempotency and the transition from at-least-once to exactly-once semantics.

First principles

From first principles

Start with the question

Why is exact counting so difficult in distributed systems?

1
Components will crash while processing data.
forced by · Hardware fails, networks partition, and deployments restart instances.
2
When a component restarts, it must retry the work it was doing when it died.
forced by · Otherwise, the data it was processing is lost.
3
The component usually cannot know if the work actually succeeded right before the crash.
forced by · The crash might have happened after the write succeeded but before the success acknowledgment was recorded.
4
Therefore, the component must retry the work, potentially doing it twice.
forced by · Safety requires retrying, and retrying risks duplication.

⇒ Therefore

Distributed systems naturally provide at-least-once delivery. Exactly-once semantics require deliberate, mathematically provable deduplication at the destination.

Architecture: The Metering Pipeline

A robust metering pipeline separates the extremely fast ingestion of raw events from the careful, deduplicated aggregation of billable usage.

1. The Ingestion Queue

When a user consumes a resource (e.g., runs a query, sends an email), the service emits a raw usage event. This event must contain a globally unique, client-generated event_id (a UUID/ULID).

The event is published to Kafka. Kafka is highly durable; once the event is acknowledged by the Kafka cluster, it is safe from loss.

2. The Metering Workers and Deduplication

The workers pull events from Kafka. Their only job is to ensure the event is counted exactly once.

To do this, they rely on idempotency. An operation is idempotent if doing it twice produces the same result as doing it once. x = 5 is idempotent. x = x + 1 is not.

Because the metering worker will inevitably crash and pull the same Kafka message twice, the database write must be idempotent.

Mental modelIdempotency via Unique Constraints

A bouncer at a club who checks ID. If you show up twice with the exact same ID card, you are denied entry the second time, but the bouncer says 'Yes, you are inside' both times to avoid confusing the person who sent you.

Every incoming event must have a unique ID generated at the absolute source.
The database must enforce a UNIQUE constraint on that ID.
If the worker tries to insert a duplicate, the database rejects it. The worker catches the DuplicateKeyException and treats it as a success.

🔔 Fires when you see

Use whenever you need exactly-once processing in a system that guarantees at-least-once delivery.

If the database being used for aggregation (like ClickHouse or Druid) does not support strict unique constraints at high insertion speeds, deduplication must happen in a separate, fast key-value store (like Redis) before the event hits the aggregator.

The tradeoff

Where to generate the idempotency key (event_id)

At the API Gateway (or Client)

+ you gain Protects against network retries. If the client's HTTP call times out and they retry, the gateway sees the same ID and deduplicates.

− you pay Requires trusting the client or gateway to generate safe, collision-free IDs.

pick when Mandatory for billing. The source of truth must generate the ID before any network hops occur.

At the Metering Worker

+ you gain Simple. The worker generates the ID when it pulls from the queue.

− you pay Fails completely. If the upstream service retried the message into the queue, the worker generates two different IDs for the same business event. The user is double-charged.

pick when Never for metering.

3. The Aggregation Database

The deduplicated events are written to an analytical database. Billing queries are analytical queries: "Sum the gigabytes transferred by User A grouped by day for the month of August."

This database must support fast time-series rollups. Often, the worker will pre-aggregate the data (e.g., summing all bytes transferred by User A in a one-minute window) before writing to the database, to reduce the row count from billions to millions.

Precision vs. Scale in Deduplication

Keeping every event_id forever in a Redis cluster to check for duplicates is impossible; the memory cost would exceed the revenue generated by the billing system.

The standard solution is a Sliding Deduplication Window.

We assume events will not arrive more than 24 hours late.
The Redis deduplication key is set with a Time-To-Live (TTL) of 24 hours.
If an event arrives after 24 hours, it is considered stale and rejected (or routed to a manual reconciliation queue).

This bounds the memory requirement to only the active events of the last 24 hours.

Failure modes

Precision loss in aggregation. Using standard floating-point numbers (FLOAT or DOUBLE) to store currency or fractional usage is a fatal error. Floating-point math introduces rounding errors (e.g., 0.1 + 0.2 = 0.30000000000000004). In a billing system processing billions of micro-transactions, these fractions compound into massive accounting discrepancies. All billing systems must use fixed-precision decimals (DECIMAL or NUMERIC in SQL, BigDecimal in code) or store values as integers (e.g., fractions of a cent).

Clock drift across servers. If the billing logic relies on the metering worker's system clock to determine which month an event belongs to, clock drift across the server fleet will cause events to be billed to the wrong month. The timestamp must be recorded at the absolute source of the event, and that timestamp must be respected throughout the entire pipeline.

Late-arriving data after invoice generation. If an invoice is generated on the 1st of the month, and a delayed event from the 31st arrives on the 2nd, what happens? A robust billing engine must support "True-up" or reconciliation phases. It never modifies a finalized invoice; it adds a line item for the late usage to the next invoice.

What to carry forward

Key points

You can now

Explain why network retries make exactly-once delivery impossible without application-level deduplication.
Design a metering pipeline that survives worker crashes without double-counting.
Manage the memory constraints of deduplication using sliding TTL windows.
Defend the choice of fixed-precision decimals over floating-point types.