Resilience Patterns and Rate Limiting

In a distributed system something is always failing: a dependency is slow, a network packet is dropped, a deploy is half rolled out. Resilience means the system keeps serving useful responses anyway, and recovers on its own. Interviewers want to hear about timeouts, retries with backoff and jitter, circuit breakers, bulkheads, load shedding and rate limiting, and how they interact. This chapter explains each and implements the core ones, with tests.

1. Failures are normal

Typical faults: a slow dependency (worse than a dead one, because callers pile up waiting), transient network errors, overload, partial deployments, bad data, resource exhaustion (threads, connections, memory), and cascading failures where one overloaded service drags down its callers.

The goals: fail fast, isolate, degrade gracefully, recover automatically, and avoid making things worse (retry storms).

2. Timeouts

Every network call needs a timeout. Without one, a hung dependency holds a thread, a connection and memory indefinitely, and a slow service quietly exhausts its callers.

• Set timeouts from the dependency's observed latency (for instance a bit above its 99th percentile) and from your own deadline.
• Use connect and read (or total) timeouts separately.
• Propagate a deadline through the call chain: if the user's request has 1 second left, a downstream call should not be given 5. Go's context, gRPC deadlines and HTTP headers carry this.
• Timeouts at each layer should be nested so the outer one is larger than the inner ones; otherwise inner retries are wasted after the caller has given up.

3. Retries, backoff and jitter

Retrying fixes transient faults, but naive retries amplify outages: if 1,000 clients retry immediately and in lockstep, a struggling service gets 1,000 more requests at the worst moment.

Rules:

1. Retry only safe operations (idempotent, or protected with an idempotency key) and only retriable errors (timeouts, connection errors, 503, 429; not 400 or 404).
2. Exponential backoff: wait $base\cdot 2^{attempt}$, capped.
3. Jitter: randomise the wait so clients desynchronise.
4. Limit attempts and the total time, and respect Retry-After.
5. Retry budgets: cap retries to a fraction of normal traffic (for example 10%), so retries cannot multiply load.
6. Avoid retrying at every layer. If three layers each retry three times, one failure becomes up to 27 requests.

import random

def backoff_delay(attempt, base=0.1, cap=5.0, rng=random.random):
    ceiling = min(cap, base * 2 ** attempt)
    return rng() * ceiling                         # "full jitter": uniform between 0 and the exponential ceiling

def call_with_retries(fn, attempts=5, sleep=lambda s: None, retriable=(TimeoutError,)):
    waits = []
    for attempt in range(attempts):
        try:
            return fn(attempt), waits
        except retriable:
            if attempt == attempts - 1:
                raise
            waits.append(backoff_delay(attempt))
            sleep(waits[-1])
    raise AssertionError("unreachable")

def flaky(attempt):
    if attempt < 3:
        raise TimeoutError("slow")
    return "ok"

result, waits = call_with_retries(flaky)
assert result == "ok" and len(waits) == 3
assert all(w <= min(5.0, 0.1 * 2 ** i) for i, w in enumerate(waits))      # each wait respects its growing ceiling

def non_retriable(attempt):
    raise ValueError("bad request")
try:
    call_with_retries(non_retriable)
    raise AssertionError("should not be retried")
except ValueError:
    pass                                                                  # a client error is not retried: retrying cannot fix it

# jitter spreads clients out: 1000 clients with the same failure retry at many different times
delays = [backoff_delay(3) for _ in range(1000)]
assert max(delays) - min(delays) > 0.5 and len(set(round(d, 3) for d in delays)) > 500

4. Circuit breaker

When a dependency is failing, continuing to call it wastes resources and slows recovery. A circuit breaker wraps calls and has three states:

• Closed: calls go through; failures are counted.
• Open: after too many failures, calls fail immediately (or use a fallback) for a cool-down period, giving the dependency room to recover.
• Half-open: after the cool-down, allow a few trial calls; if they succeed, close the circuit; if they fail, open it again.

class CircuitBreaker:
    def __init__(self, failure_threshold=3, reset_timeout=30, clock=None):
        self.threshold, self.reset_timeout, self.clock = failure_threshold, reset_timeout, clock
        self.failures, self.state, self.opened_at = 0, "closed", None

    def call(self, fn, fallback=None):
        if self.state == "open":
            if self.clock() - self.opened_at >= self.reset_timeout:
                self.state = "half-open"                       # allow one trial call
            else:
                if fallback: return fallback()
                raise RuntimeError("circuit open")
        try:
            result = fn()
        except Exception:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.threshold:
                self.state, self.opened_at = "open", self.clock()
            raise
        self.failures, self.state = 0, "closed"                # success resets the breaker
        return result

now = [0]
cb = CircuitBreaker(failure_threshold=3, reset_timeout=30, clock=lambda: now[0])
calls = []
def failing():
    calls.append(1); raise IOError("down")

for _ in range(3):
    try: cb.call(failing)
    except IOError: pass
assert cb.state == "open" and len(calls) == 3

try: cb.call(failing)
except RuntimeError: pass
assert len(calls) == 3                                          # the open circuit did not touch the failing dependency
assert cb.call(failing, fallback=lambda: "cached answer") == "cached answer"

now[0] = 31                                                      # the cool-down passed: one trial call is allowed
assert cb.call(lambda: "recovered") == "recovered" and cb.state == "closed"

Pair a breaker with a fallback: cached data, a default value, a degraded feature (recommendations replaced by popular items), or a clear error.

5. Bulkheads and isolation

Ships have watertight compartments so one leak does not sink the ship. A bulkhead gives each dependency or tenant its own pool (threads, connections, semaphore permits) so a slow dependency can exhaust only its own share.

import threading, time

class Bulkhead:
    def __init__(self, max_concurrent):
        self.sem = threading.BoundedSemaphore(max_concurrent)
        self.rejected = 0

    def run(self, fn):
        if not self.sem.acquire(blocking=False):                # no capacity: fail fast instead of queueing forever
            self.rejected += 1
            return "rejected"
        try:
            return fn()
        finally:
            self.sem.release()

slow_dependency = Bulkhead(2)
fast_dependency = Bulkhead(2)
release = threading.Event()

holders = [threading.Thread(target=lambda: slow_dependency.run(release.wait)) for _ in range(2)]
[h.start() for h in holders]
time.sleep(0.1)                                                 # give both threads time to take their permits
assert slow_dependency.run(lambda: "x") == "rejected"           # the slow dependency is saturated...
assert fast_dependency.run(lambda: "still serving") == "still serving"   # ...but the other dependency is unaffected
release.set(); [h.join() for h in holders]

6. Load shedding and backpressure

When demand exceeds capacity, serving everyone slowly makes everyone fail. Shed load deliberately:

• Reject early with 503 or 429 rather than queueing unbounded work.
• Prioritise: keep critical traffic (checkout) and drop low-priority (analytics, prefetch).
• Bound queues and use timeouts on queue wait; a request that already waited past its deadline should be dropped, since the caller has gone.
• Adaptive concurrency limits (concurrency limits that tune themselves to latency) and admission control.
• Backpressure: signal upstream to slow down (a full bounded queue, flow-control in streams, Retry-After).

7. Rate limiting

A rate limiter caps how often a client may perform an action, protecting the service from abuse, accidents and noisy neighbours, and enforcing fair use and quotas. Return 429 Too Many Requests with Retry-After, and headers such as X-RateLimit-Remaining.

Algorithms

Token bucket

class TokenBucket:
    def __init__(self, rate_per_sec, capacity, clock):
        self.rate, self.capacity, self.clock = rate_per_sec, capacity, clock
        self.tokens, self.last = float(capacity), clock()

    def allow(self, cost=1):
        now = self.clock()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)   # refill for the elapsed time
        self.last = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False

t = [0.0]
bucket = TokenBucket(rate_per_sec=2, capacity=5, clock=lambda: t[0])
assert [bucket.allow() for _ in range(5)] == [True] * 5          # a burst up to capacity is allowed
assert bucket.allow() is False                                    # then the bucket is empty
t[0] += 1.0                                                       # one second passes: 2 tokens refill
assert [bucket.allow() for _ in range(3)] == [True, True, False]
t[0] += 100                                                       # a long idle period refills only up to the capacity
assert sum(bucket.allow() for _ in range(10)) == 5

Fixed window and its boundary flaw

class FixedWindow:
    def __init__(self, limit, window, clock):
        self.limit, self.window, self.clock, self.counts = limit, window, clock, {}

    def allow(self):
        w = int(self.clock() // self.window)
        self.counts[w] = self.counts.get(w, 0) + 1
        return self.counts[w] <= self.limit

t2 = [59.9]
fw = FixedWindow(limit=5, window=60, clock=lambda: t2[0])
first = sum(fw.allow() for _ in range(5))                         # five requests just before the boundary
t2[0] = 60.1
second = sum(fw.allow() for _ in range(5))                        # five more just after it
assert first == 5 and second == 5                                 # ten requests within 0.2 seconds against a limit of 5 per minute

Sliding window log

from collections import deque

class SlidingWindowLog:
    def __init__(self, limit, window, clock):
        self.limit, self.window, self.clock, self.hits = limit, window, clock, deque()

    def allow(self):
        now = self.clock()
        while self.hits and self.hits[0] <= now - self.window:
            self.hits.popleft()                                    # forget requests outside the window
        if len(self.hits) < self.limit:
            self.hits.append(now)
            return True
        return False

t3 = [59.9]
sw = SlidingWindowLog(limit=5, window=60, clock=lambda: t3[0])
assert sum(sw.allow() for _ in range(5)) == 5
t3[0] = 60.1
assert sum(sw.allow() for _ in range(5)) == 0                      # the boundary trick no longer works: the earlier five still count
t3[0] = 120.0
assert sw.allow() is True

Where to rate limit and what to key on

• Where: the edge (CDN, API gateway) for coarse protection, plus per-service limits for fine-grained and per-endpoint control. Expensive endpoints (login, search, exports) deserve tighter limits.
• Key: user ID, API key, IP address (careful with shared NATs and spoofed headers), or a combination; tenant-level limits for multi-tenant systems.
• Distributed limiting: keep counters in a shared store such as Redis with atomic operations (INCR with expiry, or a Lua script for a token bucket) so that all instances share one count. Accept slight inaccuracy for speed (local counters synced periodically), and decide whether to fail open (allow traffic if the limiter is down) or fail closed.
• Tiers and quotas: different limits per plan; separate short-term burst limits from long-term quotas.
• Login and sensitive endpoints: limit per account and per IP to blunt credential stuffing.

8. Graceful degradation and fallbacks

Decide in advance what to sacrifice: serve cached or stale data, hide non-essential features, return partial results, queue the work for later and acknowledge, or show a clear error. Use feature flags and kill switches to turn off expensive or failing features quickly. Prefer static fallbacks (a default list) over calling another failing service.

9. Health checks and self-healing

• Liveness probe: is the process alive (restart it if not)?
• Readiness probe: can it serve traffic right now (remove from the load balancer while warming up, draining or when dependencies fail)?
• Graceful shutdown: stop accepting new work, finish in-flight requests, then exit; essential for zero-downtime deploys.
• Automatic restarts, autoscaling and replacement of unhealthy instances.
• Load balancer health checks and outlier detection eject slow or failing hosts.

10. Testing resilience

• Fault injection and chaos testing: kill instances, add latency, drop packets, fail dependencies, in staging and carefully in production.
• Load tests past expected peaks to find the saturation point and watch the failure mode.
• Game days to rehearse incident response.
• Test timeouts and fallbacks in unit and integration tests, not only the happy path.

11. Common mistakes

• No timeouts (or very long defaults) on outbound calls.
• Immediate, unlimited retries, or retries at every layer, creating retry storms.
• Retrying non-idempotent operations without protection.
• No jitter, so retries synchronise.
• Unbounded queues and thread pools that hide overload until memory runs out.
• Shared pools for unrelated dependencies, so one slow service starves the rest.
• Rate limiting only by IP behind a proxy (everyone looks like one IP) or trusting X-Forwarded-For blindly.
• Health checks that report healthy while dependencies are down, or that check so deeply they cause outages themselves.
• Never testing the failure paths.

12. Practice questions

1. Why are timeouts essential, and how do you choose them?
2. Design a retry policy for a payment call. What must be true for retries to be safe?
3. Explain the three states of a circuit breaker and what a fallback might be.
4. What is a bulkhead and what problem does it solve?
5. Compare token bucket, fixed window and sliding window rate limiters. Which allows bursts?
6. How would you implement a rate limiter shared across 20 servers?
7. A downstream service is slow and your service is falling over. What happens, and how do you prevent it?
8. What is load shedding and how do you decide what to drop?