API & Backend
Idempotent Webhooks with Node.js and PostgreSQL
An idempotent webhook handler produces the same durable result when the provider delivers the same event more than once. The practical pattern is to verify the signature over the raw request body, claim a provider-supplied event ID in a database transaction, apply local state changes atomically, and return a successful response for already-completed events.
This guide builds that flow with Node.js, Express, and PostgreSQL. It also handles concurrent duplicates, failed processing, slow side effects, replay protection, and observability. The example is provider-neutral: replace header names and signature construction with the exact contract documented by your webhook producer.
Webhook delivery is normally at least once, not exactly once. Timeouts, lost responses, worker restarts, and provider retries all create duplicates. Idempotency is therefore not an optional optimization. It is part of the consumer’s correctness model.
Define the delivery contract before writing code
A consumer needs four provider guarantees or conventions:
- A stable event identifier that remains the same across retries.
- A signature scheme with a documented payload format and algorithm.
- A retry policy, including timeout, backoff, and maximum delivery age.
- Event ordering and versioning semantics.
Do not manufacture an event ID from the body unless the provider offers none and you fully understand canonicalization. Two equivalent JSON objects can have different byte representations, while two different delivery attempts may intentionally contain an updated field. A first-class ID is safer.
Clarify what a 2xx response means. Usually it acknowledges delivery, not business completion. A consumer that starts background work and immediately returns success needs a durable queue or inbox record first; otherwise a crash between the response and enqueue loses the event.
Document the contract alongside the endpoint. Our API documentation guide covers the error and lifecycle details developers need, while the developer portal checklist helps place webhook setup, testing, and troubleshooting in a coherent journey.
Create a durable webhook inbox
The inbox table records the provider event before or with the local state transition. A unique constraint on (provider, event_id) turns the database into the concurrency gate.
CREATE TYPE webhook_status AS ENUM ('processing', 'completed', 'failed');
CREATE TABLE webhook_events (
provider text NOT NULL,
event_id text NOT NULL,
event_type text NOT NULL,
status webhook_status NOT NULL DEFAULT 'processing',
payload jsonb NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz,
attempt_count integer NOT NULL DEFAULT 1,
last_error text,
PRIMARY KEY (provider, event_id)
);
CREATE INDEX webhook_events_status_received_idx
ON webhook_events (status, received_at);Keep enough payload to replay or investigate according to your retention and privacy policy. If payloads contain regulated or sensitive data, store only the required fields, encrypt appropriately, and restrict operator access. A database is not a license to retain every event forever.
The primary key makes duplicate claims atomic. PostgreSQL’s INSERT ... ON CONFLICT chooses either the insert or conflict path even under concurrency. A preceding SELECT followed by INSERT is unsafe because two requests can both observe absence before either inserts.
The table also supports recovery. A sweeper can find old processing rows after a worker crash, while failed rows preserve a bounded diagnostic message. Store full stack traces in a protected log system rather than an unrestricted text column.
Verify the signature over raw bytes
Providers sign bytes, not the JavaScript object produced after JSON parsing. Configure the webhook route with a raw-body parser before any global JSON middleware.
import crypto from 'node:crypto';
import express from 'express';
const app = express();
const webhookSecret = Buffer.from(requiredEnv('WEBHOOK_SECRET'), 'utf8');
app.post(
'/webhooks/acme',
express.raw({ type: 'application/json', limit: '256kb' }),
handleWebhook
);
app.use(express.json({ limit: '64kb' }));
function requiredEnv(name: string): string {
const value = process.env[name];
if (!value) throw new Error(`${name} is required`);
return value;
}Assume the provider sends x-acme-event-id, x-acme-timestamp, and a hex HMAC in x-acme-signature. Verify timestamp freshness, decode the signature strictly, and compare equal-length buffers in constant time.
function verifySignature(request: express.Request): void {
const timestamp = request.header('x-acme-timestamp');
const signatureHex = request.header('x-acme-signature');
if (!timestamp || !signatureHex || !Buffer.isBuffer(request.body)) {
throw new WebhookError(401, 'missing signature fields');
}
const issuedAt = Number(timestamp);
const ageSeconds = Math.abs(Date.now() / 1000 - issuedAt);
if (!Number.isFinite(issuedAt) || ageSeconds > 300) {
throw new WebhookError(401, 'stale webhook timestamp');
}
if (!/^[0-9a-fA-F]{64}$/.test(signatureHex)) {
throw new WebhookError(401, 'invalid signature');
}
const expected = crypto
.createHmac('sha256', webhookSecret)
.update(`${timestamp}.`)
.update(request.body)
.digest();
const received = Buffer.from(signatureHex, 'hex');
if (received.length !== expected.length ||
!crypto.timingSafeEqual(received, expected)) {
throw new WebhookError(401, 'invalid signature');
}
}Timestamp validation narrows the replay window, but it does not replace event deduplication. Legitimate provider retries can occur after your freshness window; choose a tolerance compatible with the actual delivery contract. Some providers include version prefixes or multiple active signatures during secret rotation. Implement their specified parsing rather than adapting this illustrative format blindly.
Return the same generic authentication error externally. Detailed reasons belong in redacted internal telemetry. Otherwise, the endpoint becomes an oracle for signature parsing.
Claim and process the event atomically
Create a PostgreSQL pool and a handler that owns one transaction. The central rule is that the inbox claim and local database effect commit together.
import type { Request, Response } from 'express';
import { Pool } from 'pg';
const pool = new Pool({ connectionString: requiredEnv('DATABASE_URL') });
async function handleWebhook(request: Request, response: Response) {
try {
verifySignature(request);
const event = validateAcmeEvent(JSON.parse(request.body.toString('utf8')));
const eventId = event.id;
const client = await pool.connect();
try {
await client.query('BEGIN');
const claimed = await client.query(
`INSERT INTO webhook_events
(provider, event_id, event_type, payload)
VALUES ($1, $2, $3, $4)
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING event_id`,
['acme', eventId, event.type, event]
);
if (claimed.rowCount === 0) {
await client.query('ROLLBACK');
return response.status(200).json({ received: true, duplicate: true });
}
await applyEvent(client, event);
await client.query(
`UPDATE webhook_events
SET status = 'completed', completed_at = now()
WHERE provider = $1 AND event_id = $2`,
['acme', eventId]
);
await client.query('COMMIT');
return response.status(200).json({ received: true });
} catch (error) {
await client.query('ROLLBACK');
throw error;
} finally {
client.release();
}
} catch (error) {
return respondToWebhookError(response, error);
}
}validateAcmeEvent represents a runtime schema validator, not a TypeScript assertion: it must reject unknown versions, missing fields, invalid enums, oversized values, and wrong types before any database work. In particular, require a non-empty, bounded event.id from the signed JSON payload. The HMAC above authenticates the timestamp and body, not x-acme-event-id; using that unsigned header as the deduplication key would let a captured valid payload bypass deduplication by changing the header within the replay window. Use the authenticated payload ID, or follow a provider contract that explicitly signs its ID header. Type annotations disappear at runtime and cannot validate webhook JSON.
If applyEvent modifies only the same PostgreSQL database, rollback removes both the inbox row and the partial business change. The next provider retry can claim the event again. That behavior is often preferable to committing a failed row inside the same transaction, which would make a normal retry look like a duplicate.
If you need to persist failure state, do it in a separate transaction after rollback or use an attempt table. Decide whether a failed event should be retried by the provider, by an internal worker, or manually. Do not let two independent retry systems produce an unbounded storm.
Make state transitions idempotent too
Deduplicating event IDs prevents the same event from running twice. It does not solve out-of-order events or two different events that represent the same final state. Make the domain update conditional.
async function applyEvent(client: import('pg').PoolClient, event: AcmeEvent) {
if (event.type !== 'subscription.updated') return;
await client.query(
`INSERT INTO subscriptions
(external_id, status, provider_version, updated_at)
VALUES ($1, $2, $3, to_timestamp($4))
ON CONFLICT (external_id) DO UPDATE
SET status = EXCLUDED.status,
provider_version = EXCLUDED.provider_version,
updated_at = EXCLUDED.updated_at
WHERE subscriptions.provider_version < EXCLUDED.provider_version`,
[event.data.id, event.data.status, event.version, event.created]
);
}A monotonic provider version is better than comparing local receipt times. Delivery order and creation order can differ. If the provider offers only timestamps, document collision and clock assumptions. For some domains, the safest reaction to a change event is to fetch the current resource from the provider rather than reconstructing state from a possibly stale payload.
State transitions should match the domain. A shipment cannot move from delivered back to packing. A refund event should not increment a balance twice. Encode these invariants in constraints or conditional writes, not only in application branches.
PostgreSQL transactions use snapshots and locks according to the selected isolation level. Review the transaction isolation documentation when multiple events can update the same row concurrently.
Handle external side effects with an outbox
A database transaction cannot atomically commit with an email provider, message broker, or third-party API. Calling the external service before commit risks sending an email for a transaction that later rolls back. Calling it after commit risks a crash before the call.
Use a transactional outbox. Insert a side-effect command in the same transaction as the state change:
CREATE TABLE outbox_messages (
id uuid PRIMARY KEY,
topic text NOT NULL,
deduplication_key text NOT NULL UNIQUE,
payload jsonb NOT NULL,
available_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz,
attempts integer NOT NULL DEFAULT 0
);await client.query(
`INSERT INTO outbox_messages (id, topic, deduplication_key, payload)
VALUES (gen_random_uuid(), 'subscription.changed', $1, $2)
ON CONFLICT (deduplication_key) DO NOTHING`,
[`acme:${event.id}:subscription.changed`, event.data]
);A worker claims unpublished rows, sends them, and records completion. The downstream consumer or provider should also accept an idempotency key because the worker can crash after the external call but before marking the row. The outbox does not create exactly-once delivery; it gives you durable at-least-once delivery with a stable deduplication key.
Use FOR UPDATE SKIP LOCKED for parallel workers, bounded retries with jitter, a dead-letter state, and an operator replay command. Record the last error category without exposing secrets. Monitor the age of the oldest unpublished message rather than only queue length.
Choose synchronous or asynchronous processing
Process synchronously when the database work is small, predictable, and comfortably below the provider timeout. A single transaction and response are easier to reason about. Set your own deadline shorter than the provider’s deadline so the handler can fail cleanly rather than being terminated mid-commit.
Use an asynchronous inbox when work can be slow, bursty, or dependent on unreliable systems. The HTTP handler verifies, validates, and stores the event, then acknowledges it. A worker claims rows and applies the transition. The durable insert must happen before the success response.
For asynchronous processing, the uniqueness constraint remains the intake gate. Add worker leases or row locks so abandoned work can recover. Distinguish received, processing, completed, and dead_letter states. A simple processed boolean cannot explain whether work is active, retryable, or permanently rejected.
Backpressure is part of the design. Limit request body size, database connections, queue depth, worker concurrency, and calls to dependent services. A webhook burst should not exhaust the same pool used by customer-facing traffic.
Return status codes that produce the right retry
Return 2xx after the event is durably accepted or completed according to your architecture. Return 2xx for a known duplicate that already reached a safe terminal state. Providers generally interpret any 2xx as acknowledged.
Return 4xx for an invalid signature or payload that retrying cannot fix. Some providers retry every non-2xx regardless, so confirm their behavior. Avoid including sensitive validation detail in the response.
Return 5xx when a transient internal failure means the provider should retry. Do not return 200 merely to silence failures; that converts a recoverable outage into lost state. Conversely, do not return 500 after a transaction committed successfully, because the retry will add noise even if deduplication prevents damage.
Set a small response body and consistent content type. Capture the provider event ID in logs and traces, but apply your data-classification rules before treating it as non-sensitive.
Validate the event type and version before claiming work that the service cannot process. For a known but intentionally ignored event, record a bounded disposition and acknowledge it. For an unknown schema version, follow a deliberate compatibility policy: retrying forever will not teach old code a new schema, while silently accepting it can hide lost business state. Alert the owning team and preserve enough safe metadata to investigate.
Route event types through an explicit map rather than constructing function names from payload text. Reject oversized bodies before parsing, bound nested data where your schema library supports it, and keep business validation inside the transaction that owns the state change.
Test duplicates, concurrency, and recovery
A happy-path unit test proves very little. Build integration tests against PostgreSQL with these cases:
- One valid event changes state and records completion.
- The same event delivered sequentially produces one state change.
- Twenty concurrent requests with the same ID produce one state change.
- A bad signature never creates an inbox row.
- A stale timestamp follows the documented policy.
- A domain failure rolls back the inbox claim and state update.
- An older resource version cannot overwrite a newer version.
- A committed outbox message survives a worker crash.
- A provider retry after a lost HTTP response receives
2xx.
Synchronize concurrent test requests with a barrier so they reach the claim at nearly the same time. Assert database facts, not only HTTP responses. One request should own the insert; the rest should observe a duplicate without applying the transition.
Keep signed webhook fixtures and a helper that signs exact bytes. Include whitespace changes to prove the handler verifies the received representation. If your framework mutates the body first, that test will fail immediately.
Operate the webhook pipeline
Measure accepted, duplicate, rejected, failed, retried, and dead-letter events with bounded labels such as provider and event type. Track processing latency and oldest unprocessed age. Do not use event ID as a metric label.
Log one structured record per state transition with event ID, event type, outcome, attempt, duration, and trace ID. Redact payloads by default. Create alerts for sustained authentication failures, growing backlog age, repeated database errors, and a dead-letter increase.
Provide a safe replay tool that accepts explicit event IDs, shows the current state, and records the operator. Replays must run through the same idempotent domain path. A manual SQL update followed by an ad hoc script is not a dependable recovery process.
Rotate webhook secrets with overlap if the provider supports multiple signatures. Test the new secret before retiring the old one. Document firewall rules, DNS, TLS, endpoint ownership, provider configuration, and rollback in an operational runbook.
Frequently asked questions
Does a unique event ID guarantee exactly-once processing?
No. It prevents a claimed event from applying the guarded database path more than once, but external side effects can still be delivered at least once. Use transactional outboxes and downstream idempotency keys rather than claiming exactly-once behavior.
Should duplicate webhooks return an error?
Usually no. If the event already completed safely, return a successful acknowledgment so the provider stops retrying. If the earlier attempt is still processing, your response policy depends on whether concurrent processing can finish within the delivery contract.
How long should webhook event IDs be retained?
At least as long as the provider can retry or replay an event, plus an operational margin. Balance that window against storage and privacy requirements. Archive or delete payloads separately if the deduplication key needs longer retention.
Can Redis replace PostgreSQL for deduplication?
Redis can claim IDs quickly, but an expiring cache entry is not automatically atomic with your durable business state. Use it only when you can tolerate the failure modes or when a durable workflow still protects the state transition.
Make retries uneventful
The strongest webhook systems assume duplicates, reordering, crashes, and partial failures from the beginning. Verify exact bytes, let a database constraint arbitrate ownership, make domain transitions conditional, and move external effects through a durable outbox.
If you need a webhook guide, event API reference, or integration tutorial that makes these guarantees explicit, explore our API documentation service. Clear delivery semantics prevent far more integration failures than a longer endpoint list.