EvalcueAIDistributed systems interview questions
Distributed Systems Interview Questions
Distributed systems interviews test whether you can reason about partial failure and trade-offs. Prepare to explain what happens when machines disagree, messages are duplicated, networks partition, or dependencies become slower than your service target.
Consistency and data questions
Be precise about what consistency guarantee the application needs before choosing replication or storage behavior.
Strong or eventual consistency?
Strong consistency means every read sees the latest write, which costs latency and availability during failures. Eventual consistency lets replicas lag and catch up later. Decide per operation: an account balance needs the first, a like counter can live with the second.
How does leader-based replication work, and what can go wrong?
One leader accepts writes and streams them to followers. With asynchronous replication, a leader failure can lose writes that were already acknowledged, and reads from followers can be stale. Replicating synchronously to at least one follower trades latency for durability.
How do you partition data and avoid hotspots?
Hash partitioning spreads keys evenly but makes range queries expensive; range partitioning keeps ranges together but can create hot partitions. For heavily skewed keys, add a random suffix to spread writes or split the hot partition.
What do quorums give you?
With N replicas, writing to W and reading from R where W + R > N means every read overlaps at least one up-to-date replica. Lowering W or R improves latency and availability at the cost of possibly stale reads.
Messaging and failure questions
Queues and service calls introduce retries, duplicates, ordering, backpressure, and ambiguous outcomes.
At-most-once, at-least-once, or exactly-once delivery?
At-most-once can lose messages and at-least-once can duplicate them. Exactly-once delivery over a network isn't achievable in general, so systems combine at-least-once delivery with idempotent processing to get exactly-once effects.
How do you make a consumer idempotent?
Give every message a stable ID and record processed IDs in the same transaction as the side effect, or design the operation so applying it twice has no extra effect, for example setting a value instead of incrementing it.
What is a retry storm and how do you prevent it?
When a struggling service makes every client retry at once, load multiplies and the service never recovers. Use exponential backoff with jitter, cap the number of retries, and give each client a retry budget.
What is backpressure, and what do you do with messages that keep failing?
Backpressure means slowing producers down when consumers fall behind, through bounded queues or rate limits, instead of buffering forever. Messages that fail repeatedly go to a dead-letter queue so they stop blocking the rest and can be inspected.
Coordination and operations follow-ups
Experienced candidates should discuss clocks, leases, leader election, consensus at a conceptual level, split brain, failure detection, observability, and graceful degradation.
How does CAP play out in a real system?
During a network partition a system must either reject some requests to stay consistent or keep serving and risk the copies diverging. A payment ledger usually chooses consistency; a shopping cart or feed usually chooses availability and reconciles later.
How do leader election and consensus work at a high level?
Algorithms such as Raft elect a leader by majority vote and commit a log entry only once a majority has stored it, so any two majorities overlap and agree on history. Most teams rely on etcd or ZooKeeper instead of implementing this themselves.
Why can't you rely on clocks across machines?
Clocks drift and can jump when NTP corrects them, so timestamps from different machines can't safely order events. Use logical or vector clocks for ordering, and fencing tokens instead of wall-clock leases for safety.
How do you observe and contain failures?
Use distributed tracing to follow a request across services, plus metrics for latency, errors, and saturation. Contain failures with timeouts and bulkheads so one failing dependency can't take everything else down with it.
Common questions
Do I need to memorize CAP theorem?
Know what network partitions imply and how a system chooses behavior during them. Interviewers usually get more signal from a concrete consistency/availability scenario than from reciting the theorem.
What is the most important distributed-systems mindset?
Assume partial failure. A remote call can be slow, fail, succeed after your timeout, or be retried. Design protocols and state transitions so the system remains understandable under those conditions.