A queue is a promise to be slow later

Put a queue in front of a service and the graphs get calmer. Spikes that used to cause errors now cause a bit of waiting. Workers stay busy, utilization sits at a satisfying 90 %, and the error rate goes to zero. It looks like the queue solved the overload.

It didn’t. It converted it — from errors, which are loud, into latency, which is quiet. And it set a timer.

Feel the curve

The simulation is the simplest queue there is: one worker that takes on average 100 ms per job, and jobs arriving at random at a rate you control. At 6 jobs a second the worker is busy 60 % of the time. Push the rate up slowly and watch the time each job spends in the system.

Nothing much happens until about 70 %. Then everything happens. At 80 % utilization a job spends on average 5 times its service time in the system; at 90 %, 10 times; at 95 %, 20 times. Past 100 % there is no average: the queue grows without bound for as long as the overload lasts.

The formula behind the left-hand curve is the textbook result for a single random server, and it is short enough to remember:

W = S / (1 − ρ)

where W is the time in system, S the mean service time, and ρ the utilization. The 1 − ρ in the denominator is the whole story. Halving the remaining headroom doubles the wait.

Real systems are worse

That formula assumes service times and arrivals vary in a particular, well-behaved way. Real workloads are burstier: arrivals clump (a cron job, a marketing email, a retry storm), and service times have long tails (one slow query, one big tenant). Kingman’s approximation makes the effect explicit — the waiting time scales with ρ / (1 − ρ) multiplied by the average of the squared coefficients of variation of inter-arrival and service times. Double that variability term and you double the expected wait at the same utilization.

So the knee of the curve arrives earlier in production than in the simulation. A service that is “only” at 75 % can already have tail latencies that time out its callers — and timed-out callers retry, which raises utilization.

Little’s law tells you what the queue is hiding

There is one relationship that holds for any stable queue regardless of distributions:

L = λ × W

The average number of items in the system equals the arrival rate times the average time each spends there. It’s a tool for reading dashboards. If a queue holds 12,000 messages and you’re processing 200 a second, the message at the back will wait about a minute. If your SLO is 5 seconds, the queue depth chart has been telling you about an outage the whole time, in units nobody was reading.

This is why queue depth is the wrong alert. Depth means nothing without the drain rate. Alert on the age of the oldest message instead — it is the latency your users are experiencing, directly, and every serious queue (SQS calls it ApproximateAgeOfOldestMessage) can report it.

Unbounded queues turn overload into a crash

When arrivals exceed capacity, an unbounded queue turns the excess into memory. Latency climbs first, then memory, then something runs out — the broker, the heap, the disk — and you get the errors you were avoiding, all at once, plus a backlog to drain.

Worse, the work you eventually do is often useless. By the time a request that waited 40 seconds reaches the worker, the user who sent it has given up, refreshed, and sent another. The system is at 100 % utilization and zero goodput: perfectly busy, achieving nothing.

Backpressure: say no early

The alternative is to bound the queue and push back. Turn on Bound the queue at 20 in the simulation and push past 100 %: some jobs are rejected immediately, but every job that is accepted is served in bounded time. That’s a much better deal than slow failure for everyone.

The techniques, roughly in the order I reach for them:

• Bound every queue. Including the implicit ones — thread pools, connection pools, in-memory buffers, socket backlogs. An unbounded queue is a decision to fail later and larger.
• Reject fast and say when to come back. A 429 or 503 with Retry-After costs microseconds and gives well-behaved clients what they need to back off.
• Carry deadlines and check them at dequeue. If the caller’s deadline has passed, drop the job before doing it. Work for nobody is the purest waste.
• Prioritise. When you must shed, shed the batch export before the checkout. That requires knowing which is which before the overload, so put the priority on the message.
• Consider LIFO under overload. When a queue is badly backed up, the newest requests are the ones whose callers are still waiting. Serving them first, and letting the stale backlog time out, recovers goodput faster. Facebook described switching to adaptive LIFO under load for exactly this reason.

Size for the knee, not the average

Capacity plans usually target an average utilization — “we’re fine below 80 %”. The curve says that’s the wrong question. Ask instead: at peak, with real variance, how close does each queue get to the knee, and what happens to the callers’ timeouts when it does?

A queue is the right tool for smoothing short bursts and decoupling teams. It is the wrong tool for absorbing load you don’t have capacity for. Know which one you’re using it for, and make the queue tell you, loudly, when the second one starts.

Further reading

• Ben Maurer, Fail at Scale, ACM Queue (2015).
• Mor Harchol-Balter, Performance Modeling and Design of Computer Systems, Cambridge University Press (2013).
• Fred Hebert, Queues Don’t Fix Overload (2014).