Idempotency Keys in Distributed Systems: The Rules That Save You From Duplicates

A client POSTs to /charges. The request hits your server, the DB commits, the response is on its way — and then the client's TCP connection times out at 30s. The client retries. Now you have two charges on the same card.

This isn't hypothetical. It happens every day at scale. The only defense is that the server remembers 'I already processed this request' and returns the same response on retry. That's an idempotency key.

create table idempotency_keys (
  key           text primary key,           -- client-supplied UUID
  request_hash  text not null,              -- sha256 of method+path+body
  status_code   int not null,
  response_body jsonb not null,
  created_at    timestamptz not null default now()
);
create index idempotency_keys_created on idempotency_keys(created_at);

A background job deletes rows older than 24-48 hours. The primary key on key gives you free deduplication.

1. Extract the Idempotency-Key header. If missing, either reject (strict mode) or process normally (lax mode).
2. Compute request_hash = sha256(method + path + canonical_json(body)).
3. Try to INSERT the key with the hash inside a transaction that also does the real work. Use ON CONFLICT DO NOTHING and check whether it took.
4. If the insert succeeded (this is the first time), do the work and UPDATE the row with the response.
5. If the insert conflicted, SELECT the existing row: if hashes match, return the stored response; if not, return 422 Conflict.

async def handle(request):
    key = request.headers['Idempotency-Key']
    body_hash = sha256(request.method + request.path + canonical(request.body))
    async with db.transaction():
        existing = await db.fetch_one(
            'insert into idempotency_keys(key, request_hash, status_code, response_body) '
            'values ($1, $2, 0, $3::jsonb) on conflict (key) do nothing returning key',
            key, body_hash, '{}')
        if existing is None:
            row = await db.fetch_one('select * from idempotency_keys where key=$1', key)
            if row['request_hash'] != body_hash: return 422, {'error': 'idempotency_key_reuse'}
            if row['status_code'] == 0: return 409, {'error': 'in_progress'}
            return row['status_code'], row['response_body']
    result = await do_work(request)
    await db.execute('update idempotency_keys set status_code=$1, response_body=$2 where key=$3',
                     result.status, result.body, key)
    return result.status, result.body

What if the client retries while the first request is still running? The row exists with status_code = 0. Two options:

• Return 409 Conflict with a 'in progress, retry in a second' hint. Simple, correct, sometimes annoying for clients.
• Long-poll on the row until it completes. Better UX, more server complexity, watch for connection pool exhaustion.

Start with 409. Add long-polling only if the endpoint's normal latency is long enough that clients keep bumping into it.

• Server generates the key. Now every retry has a different key and idempotency does nothing.
• Key without body hash. A buggy client reuses a key for a different request; you happily return the wrong response.
• Storing only the key, not the response. You know it was processed, but can't tell the client what happened.
• TTL too short. A client that retries after 6 hours (yes, they exist) hits a fresh row and double-charges.
• TTL too long. The table grows unbounded, indexes slow down, cleanup jobs pile up.

Idempotency lives at the outermost boundary of a request. In a microservice architecture, that's usually the API gateway or the individual service's HTTP handler — not inside the domain logic. Domain logic should assume it's always the first attempt; the idempotency layer either lets it run or returns a cached response.

The same principle applies to background workers consuming a queue: the queue key (message ID or a client-supplied dedup key) is the idempotency key, checked before the job's real work starts.