EvalcueAISystem design problem
Design a Distributed Rate Limiter
Design a rate-limiting service that protects APIs across many application servers. The interview tests algorithm choice, distributed state, atomic updates, failure behavior, fairness, and the balance between strictness and availability.
Define what is being limited
Clarify whether limits apply per user, API key, IP, organization, endpoint, or a combination. Ask about burst allowance, request cost, response headers, regional scope, latency budget, and what should happen if the limiter is unavailable.
- Identity and policy dimensions
- Requests per second or weighted cost
- Burst behavior
- Fail-open versus fail-closed
Choose the algorithm from the requirements
Compare fixed window, sliding window log, sliding window counter, token bucket, and leaky bucket. Explain memory, precision, burst behavior, and how atomicity is maintained when many servers update the same limiter state.
- Token bucket for controlled bursts
- Sliding counters for smoother limits
- Atomic shared-state updates
- Local fast path versus centralized state
Handle distributed and multi-region behavior
At scale, discuss sharding by limiter key, Redis or another low-latency state store, local caches, clock assumptions, replication lag, cross-region limits, degraded behavior, configuration distribution, and observability.
- Key-based sharding
- Replication and regional limits
- Policy configuration rollout
- Metrics for blocked requests and limiter latency
Common questions
Which rate limiting algorithm is best for interviews?
There is no universal best algorithm. Token bucket is a common choice when bursts should be allowed, while sliding-window approaches can enforce smoother limits. State the required behavior and choose accordingly.
Why is atomicity important in a distributed rate limiter?
Multiple application servers may check and update the same limit concurrently. Without atomic operations, concurrent requests can all observe stale state and exceed the intended quota.
Should a rate limiter fail open or fail closed?
It depends on risk. Consumer APIs may prefer availability and fail open during limiter outages, while security-sensitive or expensive operations may choose stricter behavior. Explain the business trade-off.