Designing for Scale · Distributed Rate Limiter

The problem this post solves

A single server can easily restrict a user to ten requests per second by keeping a counter in memory. When you have a hundred servers behind a load balancer, and a user's requests are spread randomly across them, local memory is blind.

To enforce a global limit, the servers must share state. But if every API request requires a synchronous network call to update a shared counter, the rate limiter becomes a latency bottleneck that is more fragile than the API it is trying to protect.

This post covers how to choose the right limiting algorithm, and how to implement it across a distributed fleet without crushing the cache tier under lock contention.

First principles

The Algorithms

Before deciding where the data lives, you must decide how the math works. The four standard algorithms serve different traffic shapes.

Token Bucket

You have a bucket that holds a maximum of $B$ tokens. Every $T$ seconds, a refiller adds $R$ tokens to the bucket. Every request takes 1 token. If the bucket is empty, the request is dropped.

Leaky Bucket

Requests enter a queue (the bucket). The server processes requests from the queue at a strict, constant rate. If the queue is full, new requests overflow and are dropped. Pick when: You are protecting a legacy backend that will literally fall over if it receives more than exactly 50 requests per second. It smooths out bursts into a steady trickle.

Fixed Window Counter

Time is divided into fixed windows (e.g., 12:00:00 to 12:01:00). A counter increments for each request. At the minute mark, the counter resets to zero. The flaw: A user can exhaust their entire minute quota at 12:00:59, and then exhaust the next minute's quota at 12:01:01. The server receives twice the permitted load in a two-second burst at the boundary.

Sliding Window Log

Instead of a simple counter, keep a timestamp for every request the user makes. When a new request arrives, delete all timestamps older than the window, then count what's left. The flaw: It requires storing potentially thousands of timestamps per user in memory. It is perfectly accurate but memory-heavy.

Architecture: The Distributed Counter

We assume Token Bucket. The state must live in a centralized, fast, in-memory store like Redis.

The Naive Implementation (Race Conditions)

1. GET tokens for User A. (Returns 5)
2. If tokens > 0, decrement by 1.
3. SET tokens to 4 for User A.

If two API servers process a request for User A at the exact same millisecond, they both GET 5, and they both SET 4. One token was stolen.

The Lock Implementation (Slow)

1. Acquire distributed lock for User A.
2. GET tokens, decrement, SET tokens.
3. Release lock.

This solves the race condition but ruins performance. Redis spends all its time managing locks instead of serving counters.

The Scaled Implementation (Lua Scripts)

Redis is single-threaded. If you send it a short script, it will execute the entire script atomically. No other command can interrupt it.

The Lua script reads the current tokens, calculates the refill based on the current timestamp, decrements the token, saves the state, and returns the result in one single network hop, completely eliminating race conditions without locks.

Where should the Rate Limiter live?

For scale, the standard pattern is a hybrid. The API Gateway handles brute-force IP limiting (e.g., maximum 500 req/sec per IP) using local memory. The application middleware handles identity-based business limits (e.g., 10 API calls per minute for a free tier user) backed by Redis Lua scripts.

Failure modes

Failing closed. If the Redis instance holding the counters goes down, what happens? If the rate limiter fails closed, it blocks all traffic. The system goes down. A rate limiter should almost always fail open—if it cannot reach Redis, it allows the request through. It is better to temporarily serve too much traffic than to voluntarily drop all revenue.

Hardcoded limits. Baking "100 per minute" into the codebase requires a deployment to mitigate an attack. Limits must be dynamically configurable at runtime.

Silent drops. If a client hits the limit, they need to know why and when they can try again. Always return HTTP 429 Too Many Requests and include X-RateLimit-Reset headers.

What to carry forward