Designing for the six-hour outage
"It is easy to build microservices that recover from a three-second network blip. Surviving an entire physical location losing power or internet for six hours during prime service requires completely different telemetry, queue backpressure, and recovery playbooks."
In a cloud-native whiteboard diagram, high availability is drawn with multiple availability zones and auto-scaling groups. But when your backend controls physical hardware — such as 500+ kitchen display screens and POS registers in restaurants — the failure modes are grounded in concrete physical reality:
- A backhoe cuts a municipal fiber trunk down the road from the restaurant during peak dinner service.
- A blown electrical transformer plunges an entire downtown commercial strip into darkness for four hours.
- A rogue DHCP router reclaims IP addresses on kitchen monitors midway through Friday night service.
When the hardware reboots or reconnects six hours later, your backend must not choke, drop orders placed via mobile apps in the interim, or corrupt inventory counts.
1. The Single Heartbeat Pipeline
The first casualty in an outage is accurate operational awareness. Prior to our architectural overhaul at AIO, device presence was fractured: the POS wrote to one database table, the kitchen screens wrote to another, and mobile terminals maintained their own heartbeat thresholds. Support teams were overwhelmed by false disconnect alerts.
We unified all hardware heartbeats into an event-driven telemetry pipeline:
[Hardware Device]
│ (MQTT Keep-Alive, 15s interval)
▼
[AWS IoT Core]
│ (Rule Action Engine)
▼
[AWS SQS Buffer Queue]
│ (Batch consumer worker)
▼
[Redis Hash: active_device_presence] + [DynamoDB DR Snapshot]This pipeline decouples telemetry ingestion from application servers. AWS IoT Core handles high-concurrency connection pools. SQS smooths out traffic bursts. Redis provides sub-millisecond status lookups for dispatch routing, while DynamoDB stores an asynchronous cold copy with Time-To-Live (TTL) expiration for disaster recovery.
2. Exponential Backoff Ceilings & Retries
If an order cannot be delivered because a kitchen display is offline, how often should you retry?
A naive implementation retries every five seconds. If a restaurant goes offline for six hours, thousands of jobs accumulate, hammering the socket connection hundreds of thousands of times and exhausting server file descriptors.
We engineered a calibrated Exponential Backoff with Full Jitter schedule implemented directly in BullMQ:
| Attempt | Base Delay | Cumulative Elapsed Time | Action |
|---|---|---|---|
| 1–3 | 1s, 5s, 15s | < 1 minute | Transient Wi-Fi blip recovery |
| 4–6 | 1m, 5m, 15m | ~21 minutes | Local router reboot |
| 7–9 | 30m, 1h, 2h | ~3.5 hours | Extended brownout / ISP maintenance |
| 10 (Max) | 2.5h | 6 hours total | Dead-letter queue & operational audit |
By adding a random jitter multiplier between 0.8 and 1.2 to each backoff interval, we eliminate synchronized retries, guaranteeing that when the restaurant power returns, reconnecting clients do not collapse the Redis broker.
3. Recovery Replay and Edge Queue Drain
When a hardware display reconnects after a long outage, orders cannot simply flood in uncontrolled. Hundreds of accumulated tickets rendered simultaneously would crash the client DOM or trigger an endless loop of kitchen alert chimes.
The client requests a Delta Catch-up Stream:
- The client connects and transmits its last confirmed watermarked sequence ID.
- The server queries PostgreSQL for missed tickets created during the gap.
- The client receives tickets in discrete, rate-limited chunks (e.g. 10 tickets per batch) with the audio alert silenced for historical orders older than 45 minutes.
- Active current orders are highlighted with visual badges indicating elapsed preparation delay.