Webhooks That Don't Lie: Idempotency, Signatures, Replay, and a Playbook

1. Verify the signature before touching the payload. Use timing-safe compare. Reject with 401 on failure.
2. Reject clock skew — parse the timestamp header, allow ±5 minutes. Stops replay attacks cheaply.
3. Store, then process — write the raw payload to a table with an idempotency key, then enqueue. The 200 is a receipt, not a completion.
4. Make processing idempotent — same event delivered twice must produce the same result. Use unique constraints, not ON CONFLICT DO NOTHING without care.

Every mature provider (Stripe, GitHub, Shopify, Slack) signs webhooks with HMAC-SHA256 over the concatenation of timestamp + body. The exact separator differs; read the docs, then wrap it in one helper.

import { createHmac, timingSafeEqual } from "crypto";

export function verifySignature(opts: {
  secret: string;
  timestamp: string;
  body: string;   // RAW body, not JSON.stringified
  signature: string;
  toleranceSec?: number;
}) {
  const tol = opts.toleranceSec ?? 300;
  const nowSec = Math.floor(Date.now() / 1000);
  const ts = Number(opts.timestamp);
  if (!Number.isFinite(ts) || Math.abs(nowSec - ts) > tol) {
    return { ok: false, reason: "stale" as const };
  }
  const expected = createHmac("sha256", opts.secret)
    .update(opts.timestamp + "." + opts.body)
    .digest("hex");
  const a = Buffer.from(opts.signature);
  const b = Buffer.from(expected);
  const ok = a.length === b.length && timingSafeEqual(a, b);
  return ok ? { ok } : { ok: false, reason: "bad_signature" as const };
}

Providers retry aggressively — Stripe up to 3 days, GitHub 24 hours, Shopify 48 hours. If your handler does the business work inline and one downstream call fails, you'll either return 500 (and get retried) or 200 (and silently lose the event). Neither is acceptable.

The fix is boring and universal:

1. Verify signature.
2. Insert raw payload into webhook_events with an idempotency key on the provider event ID.
3. Enqueue a background job with the row ID.
4. Return 200.

create table webhook_events (
  id text primary key,             -- provider event id, e.g. "evt_1P..."
  provider text not null,
  event_type text not null,
  payload jsonb not null,
  signature text not null,
  received_at timestamptz default now(),
  status text default 'received',  -- received | processing | done | failed
  attempts int default 0,
  last_error text,
  processed_at timestamptz
);
create index on webhook_events (status, received_at);
create index on webhook_events (provider, event_type);

export async function handleWebhook(req: Request) {
  const raw = await req.text();
  const sig = req.headers.get("x-signature") ?? "";
  const ts  = req.headers.get("x-timestamp") ?? "0";

  const verdict = verifySignature({ secret: SECRET, timestamp: ts, body: raw, signature: sig });
  if (!verdict.ok) return new Response(verdict.reason, { status: 401 });

  const event = JSON.parse(raw) as { id: string; type: string };
  const inserted = await db.query(
    `insert into webhook_events (id, provider, event_type, payload, signature)
     values ($1, $2, $3, $4::jsonb, $5)
     on conflict (id) do nothing
     returning id`,
    [event.id, "stripe", event.type, raw, sig],
  );

  if (inserted.rowCount > 0) {
    await queue.publish("webhook.received", { id: event.id });
  }
  return new Response("ok"); // 200 whether new or duplicate
}

Application-level idempotency ("check then insert") races itself under retry storms. Push the invariant into a unique constraint and let the database refuse duplicates atomically.

• Idempotency key = provider event ID for inbound webhooks.
• For internal effects (charges, emails), derive a key like {event_id}:{action} and unique-index the effects table on it.
• Wrap the whole processing step in a transaction: read event → apply effects → mark done. Any failure rolls back and the event is safely retried.

create table webhook_effects (
  key text primary key,           -- e.g. "evt_1P...:charge_customer"
  event_id text references webhook_events(id),
  action text not null,
  result jsonb,
  created_at timestamptz default now()
);

Every event is already a row. Add a small UI that lists failed events with filters (provider, type, date, error class), a diff of the payload, and a one-click Replay button that re-enqueues the row after resetting status='received' and attempts=0.

In production, roughly half of all webhook incidents follow the same script: an upstream bug ships → events fail → the bug is fixed → ops replays the DLQ → the world is consistent again. Without a replay UI, that last step becomes an all-hands SQL exercise at 2am.

A well-behaved consumer retries with exponential backoff + jitter, capped by a max attempts (I default to 8). Classify errors first:

function nextDelayMs(attempt: number) {
  const base = 500;
  const cap = 30_000;
  const upper = Math.min(cap, base * 2 ** attempt);
  return Math.floor(Math.random() * upper); // full jitter
}

• Stripe — signs timestamp.body; header Stripe-Signature contains multiple key=value pairs.
• GitHub — signs the raw body only; header is X-Hub-Signature-256 prefixed with sha256=.
• Shopify — HMAC in X-Shopify-Hmac-Sha256, base64-encoded (not hex).
• Slack — signs v0:timestamp:body and the header is X-Slack-Signature prefixed with v0=.
• Twilio — signs the full URL + sorted params; if you're behind a proxy, reconstruct the URL from headers or verification fails silently.

The traditional loop — ngrok, click through the provider dashboard, wait — is slow and flaky. Two better options:

1. Cloudflare Tunnel for a stable named URL you register once with the provider.
2. Hookdeck or Convoy in front of your provider: they queue events, replay them on demand, and fan out to any environment (localhost, staging, prod) with history and retries built in.

Then commit a fixtures/ folder of recorded real events (redacted of PII) so you can run the handler under Vitest without any network at all.

import fixture from "../fixtures/stripe/checkout.session.completed.json";
import { handleWebhook } from "@/webhooks/stripe";
import { signFixture } from "@/test/webhook-signer";

test("checkout.session.completed inserts, enqueues, returns 200", async () => {
  const { body, headers } = signFixture(fixture);
  const res = await handleWebhook(new Request("http://x", { method: "POST", body, headers }));
  expect(res.status).toBe(200);
  const row = await db.oneOrNone("select * from webhook_events where id=$1", [fixture.id]);
  expect(row?.status).toBe("received");
});

• Metrics per provider + event_type: received, duplicate, verified, failed, replayed.
• Latency histogram from received_at → done_at. Alert on p95 > 30s.
• DLQ size — alert on any non-zero DLQ that hasn't drained in 15 minutes.
• Signature failures — a small baseline is normal (scanners); a spike means a secret rotation happened without updating you.