At-least-once delivery is easy; exactly-once is a product decision

Every backend engineer reaches a point in their career where someone in product management asks for "guaranteed exactly-once message delivery." It sounds completely reasonable on a slide deck: an order is submitted, so the kitchen display should ring exactly once; a card is swiped, so the merchant ledger should debit the balance exactly once.

The reality of physical distributed networks is that exactly-once message delivery over an unreliable network is impossible. Packets drop, Wi-Fi radios switch access points, TCP timeouts trigger before an acknowledgment returns, and mobile devices brown out mid-flight.

1. The Illusion of Exactly-Once

When you dispatch an event over Server-Sent Events (SSE) or a WebSocket, you can guarantee that the event was pushed into the TCP buffer. What you cannot guarantee is whether the application process on the receiver executed the handler before crashing, or whether the ACK packet sent by the receiver made it back across the cell tower.

If you do not receive an acknowledgment, you have two choices:

• At-most-once: Do not retry. You eliminate duplicates, but you accept message loss.
• At-least-once: Retry until confirmed. You eliminate loss, but you guarantee duplicate arrivals.

In financial ledgers and kitchen display hardware, silent message loss is unacceptable. Therefore, you must embrace at-least-once delivery and shift the burden of deduplication to the consumer.

2. Idempotency Keys as a First-Class Primitive

An operation is idempotent if executing it multiple times produces the identical side effect as executing it once. To achieve this, every event must carry an immutable, client-generated or ingress-generated Idempotency Key (typically a UUID v7 or deterministic hash of the business intent):

interface OrderEventPayload {
  idempotencyKey: string;     // hash(orderId + revision + storeId)
  eventId: string;
  ticketNumber: number;
  items: Array<{ id: string; qty: number; notes: string }>;
  dispatchedAt: string;       // ISO-8601 UTC
  retryAttempt: number;
}

When the receiver processes this payload, it records the idempotency key in a local persistent cache (such as an IndexedDB key-value store or PostgreSQL unique constraint table) within the same local transaction that renders the UI or updates the ledger.

If a retry packet arrives with an already-committed key, the consumer discards the side effect (the kitchen audio chime does not sound a second time; the credit card is not debited again), re-emits the exact ACK message, and exits cleanly.

3. The Explicit Device-ACK Protocol

A frequent antipattern in event architectures is marking a queue job as complete the moment the server pushes bytes to the socket. This confuses network dispatch with application consumption.

In the AIO dispatch rewrite, we established a strict contract:

1. The server retains the BullMQ job in an active state.
2. The ticket is pushed over the SSE channel with an incrementing attempt counter.
3. The client receives the ticket, renders it to the screen, writes the key to local disk, and sends a dedicated HTTP POST /api/v1/orders/:id/ack.
4. Only upon receiving this ACK does the backend BullMQ worker mark the job as completed and remove it from the Redis persistence set.

If no ACK is received within the timeout window (e.g. 15 seconds), the job moves back to the retry queue, triggering exponential backoff.

4. When Exactly-Once Is Actually a Product Decision

Why do we say exactly-once is a product decision? Because technical mechanisms can only deduplicate state; they cannot decide what human-facing workflows to present when edge anomalies occur.

Consider this case: A customer clicks "Submit Order" on a tablet. The internet drops before the screen shows confirmation. Did the order go through? The customer waits 30 seconds and clicks "Submit Order" again.

If the frontend generated a new idempotency key on each tap, the backend correctly processes two separate orders. If the frontend scoped the idempotency key to the draft basket session, the backend safely ignores the second tap. Deciding whether those two taps represent one double-clicked transaction or two intentional pizzas is not an algorithmic question — it is an intentional product design choice.

Engineering cannot solve ambiguity that product specifications have left undefined. Our job as senior backend engineers is to build deterministic primitives that make those product boundaries explicit, observable, and resilient under failure.