A sender sends a message and waits for an acknowledgement. The acknowledgement doesn’t come. What happened?

Maybe the message was lost on the way there. Maybe it arrived, was processed, and the acknowledgement was lost on the way back. Maybe it’s still in flight. From the sender’s side these are indistinguishable — the silence is the same — and no protocol cleverness changes that. The sender has to choose what to do without knowing which world it is in.

It has exactly two options. It can give up, accepting that the message may never have arrived: at-most-once. Or it can send again, accepting that the receiver may now see it twice: at-least-once. There is no third button marked “exactly once”. Every system that offers exactly-once semantics is at-least-once delivery with something on the receiving side that makes the second copy harmless.

Count the double charges

The simulation sends twelve payment messages across a lossy link. Start with the defaults — retries on, no idempotency key, 20 % of acknowledgements lost — and watch the receiver’s ledger.

FIG. 2 — Lost messages and lost acks
Behaviour
attempts 0 · applied 0 · duplicates 0 · lost 0 · deduped 0

Running…

Twelve charges, a link that drops messages and acknowledgements independently. Retries make delivery reliable; only the idempotency key makes it safe.

Every charge arrives — retries guarantee that — but some customers are charged twice. Each of those is a lost acknowledgement: the receiver did the work, the sender never heard, the sender tried again, and the receiver did the work again.

Now turn Retry on timeout off. The duplicates disappear and lost charges take their place. Turn retries back on and switch Idempotency key on: every charge is applied once. Duplicates still arrive — the network is exactly as unreliable as before — but the receiver recognises them and answers with the result it already has.

Which guarantee do you need?

That depends entirely on what a duplicate means to the person on the other end.

A duplicated metrics sample is noise; at-most-once is fine and cheaper. A duplicated push notification is annoying but harmless. A duplicated shipment request means two parcels on one invoice. A duplicated charge is a support ticket, a refund, and a customer who now reads every statement. The question “what delivery guarantee do we need?” is really a product question: what does a second copy cost, and who pays for it? Engineers should ask it out loud, because the default answer — whatever the queue library does — is rarely the right one.

Exactly-once effects: the idempotency key

The pattern that makes a duplicate harmless is old and simple.

  1. The client generates a key when the user forms the intent — when they press “Pay”, not when the HTTP request is sent. Every retry of that intent carries the same key.
  2. The receiver records the key and the result in the same transaction as the side effect.
  3. If the key has been seen, the receiver returns the stored result without doing the work again.

In PostgreSQL, a unique constraint does the hard part:

create table idempotency_keys (
  key        text primary key,
  response   jsonb,
  created_at timestamptz not null default now()
);
async function chargeOnce(db: Db, key: string, charge: Charge) {
  return db.transaction(async (tx) => {
    const claimed = await tx.query(
      `insert into idempotency_keys (key) values ($1)
       on conflict (key) do nothing
       returning key`,
      [key],
    );

    if (claimed.rowCount === 0) {
      // Seen before: answer with what we answered last time.
      const { rows } = await tx.query(
        `select response from idempotency_keys where key = $1`,
        [key],
      );
      return rows[0].response;
    }

    const result = await applyCharge(tx, charge);
    await tx.query(
      `update idempotency_keys set response = $2 where key = $1`,
      [key, result],
    );
    return result;
  });
}

The insert is the lock. If two retries of the same request arrive concurrently, the second insert … on conflict do nothing waits for the first transaction to commit, then finds the row and returns the stored response. There is no window where both pass a check and both charge.

Where this goes wrong in practice

The pattern is simple; the ways to break it are subtle.

  • A key per attempt. If the key is generated inside the HTTP client, every retry gets a fresh one and the receiver dedupes nothing. The key belongs to the user’s intent.
  • Check, then act, in separate steps. select to see if the key exists, then insert and charge, is a race. Two concurrent retries both see “not found”. Let the database’s uniqueness do the checking.
  • A dedup window shorter than the retry horizon. If keys expire after an hour but a client can retry after a day offline, the late retry is a new charge. The window must outlive every path that can resend.
  • Side effects outside the transaction. The charge is idempotent, but the confirmation email is sent from the same handler, before the commit. The email goes out twice; or worse, goes out for a charge that rolled back.
  • Acknowledging receipt instead of effect. A consumer that acks a message when it arrives, then crashes before applying it, has silently turned at-least-once into at-most-once. Ack after the effect is durable — not before.

Getting events out: the outbox

The email problem generalises. Whenever a state change must also produce a message — notify a device, publish an event, call a webhook — you have two systems that can’t commit atomically together. Write to the database, then publish? A crash between the two loses the event. Publish, then write? A crash leaves an event for a change that never happened.

The transactional outbox resolves it by not having two systems at commit time. In the same transaction as the state change, insert the event into an outbox table. A separate relay reads the outbox and publishes, marking rows sent only after the broker confirms. The relay may publish an event twice — it is at-least-once — so consumers carry idempotency keys too. It’s the shape I reach for whenever a pipeline must neither lose nor double anything: at-least-once everywhere, idempotent at every edge where a duplicate would cost someone something.

The promise is per side effect

“Our system has exactly-once semantics” is almost never true as a sentence about a system. It can be true as a sentence about a specific side effect — this charge, this ledger entry, this ticket on this screen — with a named key, a named store, and a dedup window that outlives the retries.

Make that promise deliberately, effect by effect, to the person who would otherwise pay for the duplicate. Everywhere else, at-least-once is fine, and cheaper than you think.

Further reading