A retry looks like kindness. A request failed, the network is unreliable, try again — what could be more reasonable? On a healthy day, nothing. On the day a dependency falls over, every one of those reasonable retries arrives at the same moment, aimed at the one component that can least afford it.

The arithmetic is unforgiving. If a service retries a failed call three times, one logical request can become four attempts. Put three such services in a chain — an API that calls a service that calls a database, each with the same polite policy — and a single failure at the bottom can become 4 × 4 × 4 = 64 attempts at the thing that is already failing. Nobody designed that number. It falls out of three sensible-looking config files.

Watch it happen

The simulation below is a server that normally handles 60 requests a second, serving 40 a second of real demand. It has comfortable headroom. Every client gives up on an attempt after half a second and allows itself up to five retries. Try a 1 s blip first, then Fail server for 3 s, and watch what happens after capacity comes back.

FIG. 1 — A retry storm
Retry policy
offered 0/s · goodput 0/s · retries 0% · status: HEALTHY
Fifty clients, 40 requests per second, a server with capacity for 60. Every policy allows up to five retries. The policies differ only in when — and whether — each retry is sent. You can switch policy while it runs.

The blip is harmless: a short burst of retries, absorbed by the headroom. The three-second outage is not. With Immediate retries, capacity returns and the server does not. The demand that caused the failure is long gone, and goodput stays pinned near zero. That is the part that surprises people in post-mortems: the trigger was three seconds long, the outage was not.

Why it doesn’t recover

Two effects compound.

First, retries raise offered load. During the outage, every request fails and is retried, and the retries fail and are retried again. When capacity returns, it faces the new traffic plus every pending retry — several times more than 60 a second.

Second, overloaded servers do less useful work, not the same amount more slowly. Requests queue past their client’s timeout, so the server spends CPU on responses nobody is waiting for. Threads contend on locks, connection pools thrash, garbage collection runs more often. The simulation models this crudely — above capacity, effective throughput falls — but every real system has some version of it.

Put them together and you get a loop: failures cause retries, retries cause overload, overload causes failures. The system has two stable states, and the outage pushed it from the good one into the bad one. This has a name, metastable failure, and the defining property is exactly what you just watched — the outage persists after its trigger is removed. You cannot fix it by restoring the thing that broke. You have to remove the load that keeps it broken.

Backoff doesn’t save you

The textbook fix is exponential backoff: wait 250 ms before the first retry, 500 ms before the second, then 1 s, 2 s, capped. Switch the simulation to Backoff and fail the server again.

It still doesn’t come back. The shape changes — the first seconds after the outage are a little gentler — but the server settles into the same collapsed state. Add jitter, which randomises each delay so clients stop retrying in lockstep, and the result is the same.

That’s not a quirk of the simulation. Backoff changes when a client sends its retries. It doesn’t change how many it sends: each failing request still makes up to six attempts. And in a system like this one — an open system, where new requests keep arriving whether or not old ones succeed — the server’s load is set by the arrival rate times the attempts per request. A failing request that waits longer before retrying is simply replaced, in the meantime, by new requests that are failing too.

Backoff is genuinely useful in closed systems, where a fixed set of clients each wait for their own previous request — a worker pool, a batch job, a single client hammering an API. There, slowing each client really does slow the total. Jitter is genuinely useful against synchronised spikes, where many clients fail at once and would otherwise retry at once. Neither one puts a ceiling on how much extra work you’ll send when everything is failing.

Budgets put a ceiling on the damage

A retry budget does. It caps retries as a fraction of real traffic. The version in the simulation is a token bucket shared by the client: every success deposits 0.1 tokens, every retry costs one, and when the bucket is empty the client fails fast instead of retrying.

class RetryBudget {
  private tokens = 10;

  constructor(private ratio = 0.1, private max = 10) {}

  recordSuccess(): void {
    this.tokens = Math.min(this.max, this.tokens + this.ratio);
  }

  // Call before each retry. false means: give up now, don't add load.
  tryAcquire(): boolean {
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
}

When things are healthy, a budget is invisible — successes keep the bucket full, so the occasional transient failure is retried as usual. When things are broken, successes stop, the bucket drains, and the client stops piling on. Retries can never exceed about 10 % of the traffic that is actually succeeding. Select + Retry budget and the server is back as soon as its capacity is — even if you switch to it while the server is stuck.

The cost is honest: during the outage, more requests fail outright instead of being retried. That is the trade. A retry budget chooses a fast, visible failure for some users over a slow, total failure for all of them. Circuit breakers make the same trade at a coarser grain.

Where the retries should live

Most of the damage in real systems comes not from any single policy but from retries at every layer. A few rules I apply in design reviews:

  • Retry at one layer. Usually the one closest to the user, which knows whether the work still matters. Inner layers fail fast and say so.
  • Bound the volume, not just the timing. A retry budget or circuit breaker, in addition to backoff and jitter.
  • Propagate deadlines. If the caller will give up in 2 seconds, a downstream retry scheduled for second 3 is pure waste. Pass the deadline with the request and check it before every attempt.
  • Only retry what is safe to repeat. A retried read is free. A retried payment is a second payment unless the operation carries an idempotency key — which is its own essay.
  • Retry on the right errors. Timeouts and 503s, yes. A 400 will be a 400 next time too.
  • Make retries visible. Graph retry attempts as their own series. In an incident, “35 % of our traffic is retries” is the single most useful sentence anyone can say.

The policy is part of the capacity plan

Capacity planning usually asks how much traffic the system must handle on its busiest day. Retry policy answers a different question: how much traffic will the system send itself on its worst day? If the answer is “six times normal, until something gives”, the busiest-day number doesn’t matter.

So when I review a retry policy, I read it as a load forecast. How many attempts, from how many layers, at what spacing, bounded by what? If the last answer is “nothing”, the policy is going to write the next outage.

Further reading