Skip to main document
MUSTAFA SHAIKH Sr. Backend Engineer · Dossier
[DOSSIER 02] · AIO PLATFORM · 2025 – PRESENT
PRODUCTION ACTIVE · ENTERPRISE HOSPITALITY

AIO — Real-Time Order Delivery & Data Integrity

"At-least-once real-time dispatch with zero financial drift under continuous edge hardware failure."

System Metric Engineering Scope Architecture Layer
Role Senior Backend Engineer Event streams & core ledger
Scale & Edge ~500 production hardware devices; load-tested to 10,000 concurrent SSE sockets KDS, POS & expo displays
Core Stack NestJS, TypeScript, PostgreSQL, Redis, BullMQ, SSE, AWS SQS & IoT Core, DynamoDB Distributed event mesh
Tolerance 1¢ financial tolerance; 6-hour disaster recovery backoff with 10 max retries Zero drift guarantee

01 // Context: High-Volume Commercial Kitchen Dispatch

SEC. 01

In commercial restaurants running on the All-in-One (AIO) platform, dozens of physical screens—Kitchen Display Systems (KDS), expedite monitors, order pickup displays, and mobile handheld waiter tablets—must coordinate meal preparation in sub-second time. When a party orders four entrees, a dropped kitchen ticket means cold food, long wait times, and direct restaurant revenue loss.

The production system suffered from three fundamental structural vulnerabilities:

  1. Ephemeral Dispatch Loss: Orders broadcast over raw SSE connections vanished permanently if kitchen tablets were moved behind stainless steel refrigerators or power cycled mid-shift.
  2. Fragmented Liveness Telemetry: Hardware status was recorded across four disparate database tables with desynchronized heartbeat timeouts, leaving support engineers unable to determine whether a device was genuinely offline or merely quiet.
  3. Financial Ledger Drift: Subtle discrepancies accumulated across daily register shifts, caused by edge cases in zero-tip credit transactions and inconsistent cash drawer reconciliation tolerances.
Kitchen Reality: Kitchen screens are mounted above fryers and grills in high-heat environments. Power cables get unplugged by accident, Wi-Fi fluctuates under kitchen microwave interference, and tablet OSs perform background updates. The backend cannot assume persistent TCP sessions.

02 // Constraints: Zero Loss & Strict Closing Ledger Balances

SEC. 02

The operational requirements in restaurant operations eliminate conventional best-effort messaging compromises:

Constraint Technical Boundary Operational Consequence
At-Least-Once Delivery Every order ticket must reach the kitchen display; duplicates must deduplicate safely at the device edge. No missing customer meals under any circumstance.
6-Hour Outage Recovery Exponential backoff retry queues must retain undelivered jobs up to 6 hours (max 10 retries). Covers complete lunch or dinner shift power cut recovery.
Single Liveness Model Consolidate 4 competing status stores into a single AWS IoT Core → SQS → Redis heartbeat pipeline. Eliminates false alert storms for support teams.
1-Cent Ledger Parity Strict reconciliation of HouseDrawer closing balances, eliminating the legacy 3-cent tolerance. Satisfies audited hospitality accounting standards.

03 // Decisions: BullMQ Persistence & Explicit Device ACK

SEC. 03

To replace fragile ephemeral broadcasts, I engineered an at-least-once delivery pipeline utilizing BullMQ job persistence, Redis Pub/Sub cross-instance routing, Server-Sent Events (SSE), and an explicit Device Acknowledgment (ACK) contract.

FIGURE 2.0 // AT-LEAST-ONCE DISPATCH & DEVICE ACK PROTOCOL LOAD: 10,000 CONCURRENT CONNS
INGESTION & PERSISTENCE ORDER INGESTION (NESTJS) Idempotency Validation BULLMQ QUEUE ENGINE Durable Job Stream 6h Exponential Backoff (10x) POSTGRESQL STORE Immutable Transactional Log CROSS-INSTANCE ROUTING REDIS PUB/SUB CLUSTER Instance Sharding & Broadcast SSE Subscription Map AWS IOT CORE HEARTBEAT SQS → Redis Single Liveness DYNAMODB DR SNAPSHOT Async Disaster Recovery HARDWARE EDGE (~500) SSE KITCHEN DISPLAY (KDS) Renders Ticket & Audio Tone Deduplicates on Client ID DEVICE ACK: CONFIRMS RENDER → DELETES JOB FROM QUEUE
Figure 2.0: At-least-once order dispatch pipeline. The BullMQ worker retains the job in Redis until the kitchen terminal renders the UI ticket and fires an explicit HTTP Device ACK back. If connection drops, exponential backoff retries preserve ticket delivery for up to 6 hours.

Additional major engineering decisions delivered:

  • Complete Giftcard Subsystem Rewrite in 10 Days: Rebuilt a legacy giftcard service in 10 days, designing a normalized relational schema supporting sell, reload, transfer, and multi-tender POS redemptions with strict SOLID boundaries.
  • DynamoDB DR Snapshots: Asynchronous chunked writes capture live device state into DynamoDB with TTL cleanup, providing an independent failover path during Redis cache purges.
  • N+1 Query Elimination in Event Processing: Replaced iterative queries in SQS workers with chunked batch lookups, reducing AWS RDS CPU utilization during lunch rushes.
ACK Timing Tradeoff: Acknowledging after SSE push creates false positives if the client disconnects before rendering. Acknowledging after DOM render introduces potential duplicates if the ACK drops on return. We deliberately chose the latter, paired with deterministic ticket deduplication on the display.

04 // Failure Modes: Hardware Dropouts & Retry Thundering Herds

SEC. 04

Deploying distributed systems to 500 physical restaurant sites means every network fault scenario will happen simultaneously during peak weekend rush:

Failure Scenario Immediate Production Risk Engineered Mitigation & Recovery
6-Hour Power Blackout Kitchen display drops for entire shift; hundreds of queued tickets accumulate. BullMQ exponential backoff with max 10 retries retains jobs. When device re-connects, tickets replay sequentially without overwhelming edge memory.
Reconnection Thundering Herd Multiple terminals reconnect simultaneously, flooding backend SSE server nodes. Applied randomized jitter to client reconnect intervals and Redis Pub/Sub backoff intervals, preventing CPU spikes.
Lost Device ACK Packet KDS receives and displays ticket, but the return ACK packet drops on unstable Wi-Fi. The server retries. Upon receiving the duplicate ticket ID, the KDS deduplicates in local memory and re-emits the ACK immediately without ringing twice.
Closing Cash Drawer Drift Zero-tip card transactions and rounding discrepancies created end-of-day register closing mismatches. Tightened ledger reconciliation tolerances from 3 cents to 1 cent and resolved card transaction rounding math, restoring audited balance parity.

05 // Outcome: Production Load Testing & Operational Metrics

SEC. 05

The rewritten dispatch and telemetry infrastructure is running in production across ~500 restaurant hardware endpoints. The dispatch architecture was verified through synthetic load testing up to 10,000 concurrent SSE connections without message loss:

Hardware Devices ~500 Kitchen displays & POS
Load Tested 10K Concurrent SSE connections
Auto-Recovery 6h Exponential backoff window
Giftcard Rewrite 10d Full domain replacement
Precision Audit 1¢ Tolerance (from 3¢)

By pairing durable BullMQ message queuing with explicit client ACK protocols and consolidated AWS IoT telemetry, we transformed an intermittent restaurant POS setup into a resilient edge computing network that survives real kitchen stress.