Somewhere in your system there is a line of code that decides a machine is dead. It might be a load balancer’s health check, a cluster membership protocol, a device dashboard turning a tile red, or a lock that expires so another worker can take over. Whatever form it takes, it is making a guess — and it is wrong more often than anyone writes down.
The reason is fundamental, not an engineering shortfall. In an asynchronous network, where messages can be delayed by an unknown amount, there is no way to distinguish a process that has crashed from one that is merely very slow. A node in a 3-second garbage-collection pause looks exactly like a dead one for 3 seconds. A congested link makes a healthy node look like a flapping one. The only thing a failure detector can do is decide how long to wait before it stops believing — and every choice of “how long” is a trade.
Pick a timeout
The simulation below has a node sending a heartbeat every second over a network with some jitter. At 40 seconds it stalls for 3.5 seconds in a stop-the-world pause. At 100 seconds it really dies. Your job is to choose a timeout that catches the death quickly without ever declaring the node dead while it was alive.
Short timeouts notice the death almost immediately — and also mourn the pause, and with high jitter, a few ordinary late heartbeats too. Lengthen the timeout until the false alarms stop and look at the detection delay you’ve bought. The trade-off chart makes the shape explicit: as one line falls the other rises, and there is no setting where both are zero. There never is.
What a false alarm costs
The right point on that curve depends on what happens when the detector is wrong, and that is a property of your system, not of the detector.
If a false alarm only colours a dashboard tile amber for a few seconds, be aggressive. If it triggers a failover — a replica promoted, a partition rebalanced, a fleet of clients reconnecting to a new host — then each false alarm is a small self-inflicted outage, and it may cascade: the rebalance adds load, the load causes pauses, the pauses cause more false alarms. Over-eager detectors are a classic way for a cluster to keep “failing over” while every node is healthy.
So the first question in any review of a health check is not “how fast does it detect?” It’s “what does it do when it’s wrong, and how often is it wrong?”
Adapt to the network you actually have
A fixed timeout is a guess about network behaviour, made once, at configuration time. Networks don’t hold still: a detector tuned in a quiet data centre will cry wolf on a busy one, and one tuned for a busy network will be needlessly slow in a quiet one.
Accrual failure detectors replace the boolean with a level of suspicion. The φ (phi) accrual detector keeps a window of recent heartbeat inter-arrival times, models their distribution, and asks: given how heartbeats have been arriving lately, how unlikely is it that we have heard nothing for this long? φ is that improbability on a log scale — φ = 1 means roughly a 10 % chance the silence is normal, φ = 8 means roughly one in a hundred million. You set a threshold on φ instead of on seconds, and the effective timeout stretches automatically when the network gets noisy. Cassandra and Akka both use variants of it.
Switch the simulation to Adaptive (φ) and raise the jitter. The detector widens its own margin as heartbeats become irregular, so it holds steady where a fixed timeout set for a quiet network starts to flap. It is still a trade-off curve — just one that tracks the network instead of a guess about it.
Two states are not enough
Most of the liveness bugs I’ve worked on were not detector-maths problems. They were state-machine problems.
Separate suspect from dead. A missed heartbeat should move a node to suspect — stop routing new work to it, keep its existing work. Only sustained silence moves it to offline, which is when you reassign work. Most transient blips resolve inside suspect without anyone noticing.
Add hysteresis on the way back. A node that flaps between online and offline every few seconds is worse than either state. Require several consecutive heartbeats before returning to online.
type Liveness = 'online' | 'suspect' | 'offline';
function nextState(state: Liveness, silentFor: number, streak: number): Liveness {
// silentFor: seconds since the last heartbeat
// streak: consecutive heartbeats received on time
if (silentFor > OFFLINE_AFTER) return 'offline';
if (silentFor > SUSPECT_AFTER) return state === 'offline' ? 'offline' : 'suspect';
if (state !== 'online' && streak < RECOVER_STREAK) return state;
return 'online';
}Have one source of truth. When the API gateway, the job scheduler and the admin dashboard each infer liveness their own way — one from connection state, one from last job completion, one from a cached column — they will disagree, and every disagreement becomes a bug report that nobody can reproduce. Compute liveness in one place, publish the transitions as events, and let everything else subscribe.
Send more than a pulse. Put a sequence number in each heartbeat so the receiver can tell “lost” from “late”, and measure intervals on the receiver’s monotonic clock, never by comparing timestamps from two machines.
When being wrong means two leaders
There is one case where no detector is good enough. If declaring a node dead lets another node take over its role — a lock holder, a primary, a leader — then a false alarm creates two of them, and the old one doesn’t know it has been replaced until it’s too late. Its stale writes land after the new leader’s.
No timeout fixes this, because the old node can pause for longer than any timeout. The fix is on the resource side: every grant of the role comes with a monotonically increasing fencing token, and the storage rejects writes carrying a token older than the newest it has seen. The detector is then allowed to be wrong, because being wrong is no longer dangerous.
That is the general lesson. You can’t build a failure detector that is never wrong. You can build a system in which its being wrong is cheap.
Further reading
- Naohiro Hayashibara et al., The φ Accrual Failure Detector, SRDS (2004).
- Tushar Chandra and Sam Toueg, Unreliable Failure Detectors for Reliable Distributed Systems, Journal of the ACM (1996).
- Martin Kleppmann, How to do distributed locking (2016).