Webhook endpoints are easy to write and easy to write badly, and the bad version works fine right up until traffic, a slow database, or a redelivery storm exposes it.

The four properties below are not novel. They are just the ones that get skipped, and each one maps to a specific failure that is painful to debug after the fact.

1. Verify the signature before you parse the body

Any endpoint that accepts events must confirm the sender. That means computing an HMAC over the raw request body and comparing it in constant time.

The raw part is the trap. Most frameworks parse JSON before your handler runs, and JSON.stringify(req.body) is not the bytes that were signed — key order, whitespace, and unicode escaping all differ.

import { createHmac, timingSafeEqual } from "node:crypto";

export async function POST(request: Request) {
  // Read the body as text first. Signature is computed over these exact bytes.
  const raw = await request.text();
  const signature = request.headers.get("x-signature") ?? "";
  const timestamp = request.headers.get("x-timestamp") ?? "";

  // Bind the timestamp into the signed payload so a captured request cannot be
  // replayed later with the same signature still valid.
  const expected = createHmac("sha256", process.env.WEBHOOK_SECRET!)
    .update(`${timestamp}.${raw}`)
    .digest("hex");

  const a = Buffer.from(signature);
  const b = Buffer.from(expected);
  if (a.length !== b.length || !timingSafeEqual(a, b)) {
    return new Response("invalid signature", { status: 401 });
  }

  // Reject anything outside a narrow window, even if the signature is valid.
  const age = Date.now() - Number(timestamp) * 1000;
  if (!Number.isFinite(age) || age > 5 * 60_000) {
    return new Response("stale", { status: 401 });
  }

  const event = JSON.parse(raw);
  // ...
}

Use timingSafeEqual, not ===. The comparison is short and the difference feels theoretical, but it costs one import.

Failure prevented: anyone who learns your endpoint URL can inject decision events into your system.

2. Acknowledge fast, process later

The handler’s job is to durably record that the event arrived and return. It is not to do the work.

const event = JSON.parse(raw);

// One write. Then get out.
await events.insert({
  id: event.id,
  type: event.type,
  payload: raw,
  receivedAt: new Date(),
  status: "pending",
});

await queue.enqueue({ eventId: event.id });

return new Response(null, { status: 204 });

Everything interesting — updating the application, notifying the consumer, writing to the LOS — happens in a worker reading that queue.

The reason is delivery semantics. A sender retries on timeout. If your handler takes eleven seconds because it is doing real work, and the sender’s timeout is ten, you will receive that event again while you are still processing it the first time. Now you have concurrency you did not design for, on a code path that assumed it was alone.

Failure prevented: slow downstream dependencies turning into duplicate processing and retry storms.

3. Deduplicate on the event id

Webhook delivery is at-least-once. Assume every event will arrive twice, because eventually one will.

The cheapest defence is a unique constraint on the event id and an insert that tolerates conflict:

create table webhook_events (
  id            text primary key,
  type          text not null,
  payload       jsonb not null,
  received_at   timestamptz not null default now(),
  processed_at  timestamptz
);
const { rowCount } = await db.query(
  `insert into webhook_events (id, type, payload)
   values ($1, $2, $3)
   on conflict (id) do nothing`,
  [event.id, event.type, raw],
);

// Zero rows means we have already seen this one. Acknowledge and stop.
if (rowCount === 0) return new Response(null, { status: 204 });

Note that the deduplication key is the event id, not the resource id. Two distinct events about the same application are not duplicates, and keying on the application would drop the second one.

Failure prevented: the same state transition applied twice — a duplicate notification at best, a double-counted decision at worst.

4. Tolerate out-of-order arrival

Retries and parallel delivery mean event 5 can arrive before event 4. If your handler applies whatever it receives, a stale event can overwrite a newer state.

Carry a monotonic value on the resource — a sequence number or a version — and refuse to move backwards:

await db.query(
  `update applications
      set status = $1, version = $2
    where id = $3
      and version < $2`,
  [event.data.status, event.sequence, event.data.applicationId],
);

If the update matches no rows, a newer event already landed. That is not an error and should not be retried — it is the guard doing its job. Log it and acknowledge.

Failure prevented: an application flipping from approved back to in_review because a delayed retry of an older event arrived last.

The part that is actually hard

None of the four is difficult. What is difficult is the fifth thing, which is operational rather than technical: knowing when events stopped arriving.

A handler that fails loudly is fine. A sender that goes quiet — because a certificate expired, or a firewall rule changed, or the endpoint started returning 401 and the retries exhausted — produces no errors on your side at all. Everything looks healthy. The queue is empty because nothing is arriving.

Alert on the absence of expected events, not just on the presence of failed ones. For most integrations a simple “no events of type X in N minutes during business hours” check catches the entire class.

Endpoint names, headers, and payload fields above are illustrative. The webhook interface, signing scheme, and retry policy for your integration are confirmed during technical discovery.