Webhook Reliability and Security

1. First screen: the dangerous success case

A webhook receiver can return 200 OK, log a plausible JSON object, and still be wrong in several independent ways. The payload may have been forged. A valid signed delivery may have been replayed. The same business event may be delivered under another delivery attempt. The handler may perform a durable side effect and crash before it records completion. A later event may arrive before an earlier one. None of those failures is disproved by a green HTTP status.

This lesson therefore treats a webhook receiver as a boundary with five separate contracts: authenticity, secret handling, duplicate/replay control, acknowledgement versus processing, and ordering/recovery. The lab uses a deterministic local billing-provider fixture. It does not depend on a real provider or outbound network. Your final evidence artifact is Webhook Authenticity, Replay & Idempotency Record v1.

The core rule is deliberately narrow: verify provider-defined authenticity over the bytes the provider says were signed, then process the verified message under explicit duplicate, durability and ordering policies. Do not turn one provider example into a universal webhook standard.

2. Route orientation

Foundation route: keep the five contracts visible as separate rows. First prove that the signature check uses the exact raw body. Next distinguish a delivery identifier from the business event identifier. Then make one side effect duplicate-safe. Only after those are stable should you reason about acknowledgement and reordering.

Repair route: begin from the failed evidence row. If a valid delivery fails after JSON parsing, inspect the raw-byte boundary. If duplicates change business state twice, inspect the idempotency identity. If a crash creates a lost or repeated effect, inspect the write ordering. If an old event overwrites a new state, inspect the business version/reconciliation rule.

Experienced route: you may skip basic HMAC syntax, but you may not skip the mutation probes or the evidence record. Your task is to state the provider-specific assumptions, define the durability boundary, and explain what remains unproven by the local fixture.

3. Mental model: one request, several identities

A reliable receiver should not collapse every identifier into one field. A useful model is:

raw bytes + provider signature + delivery ID → authenticity gate → parse → business event ID/version → duplicate-safe durable effect → delivery completion mark → transport acknowledgement

The delivery ID identifies one delivery attempt or provider delivery record. A business event ID identifies the semantic event you do not want to apply twice. A business object version or equivalent domain marker helps decide whether a late event is stale, current, or evidence of a gap. Providers differ; some expose all of these concepts, some expose fewer, and names differ.

Authenticity answers: “Does this message match the provider-specific verification contract and secret material?” It does not answer: “Have I processed it before?” HMAC is a message-authentication construction, not a replay database. Duplicate control therefore needs state outside the MAC computation.

Transport acknowledgement answers another question: “What response will the sender observe for this delivery?” It does not automatically prove that downstream business work is durable. Your design must choose when a success response becomes safe.

4. Foundation layer: raw bytes, HMAC, and secrets

For a GitHub-style example, the receiver gets a signature header such as X-Hub-Signature-256. The expected value is derived from a shared secret and the payload contents. The verification input must preserve the payload bytes required by the provider contract. If middleware parses JSON and then serializes it again before verification, whitespace, escaping, or encoding changes can make a valid signature fail. Worse, a team can accidentally verify a representation different from the one the provider signed.

A safe sequence is: capture the raw request body, read the provider signature header, calculate the expected MAC with the configured secret, compare values with an appropriate constant-time comparison, reject on mismatch, and only then trust/parse the body for business processing.

Secrets are configuration, not source literals. Rotation is also a contract, not merely “replace the string.” A bounded rotation design can accept a current secret and a previous secret for a limited overlap window if the provider supports that transition, while signing/verifying new configuration according to provider semantics. Record which secret generation matched, without logging the secret itself.

Do not generalize the GitHub header name or algorithm to every provider. The lesson uses GitHub as a concrete provider example because its current documentation is explicit about X-Hub-Signature-256; the architecture rule is to implement the provider-defined verification contract.

5. Deep explanation: duplicate safety and the crash window

A provider can redeliver after a timeout, network failure, or explicit redelivery action. Even if the second HTTP request is bit-for-bit authentic, applying the business effect again may be wrong. The receiver therefore needs an idempotency strategy at the business boundary.

The local fixture records both delivery completion and applied business event identity. Consider a payment event evt-200 delivered as delivery-A. The handler verifies authenticity, applies evt-200, then crashes before marking delivery-A complete. The provider retries delivery-A. If the system only deduplicates by “processed delivery IDs,” the second attempt sees no completion record and can apply the payment twice. The business event identity is the stronger guard for the effect.

The repaired flow is: verify; parse; atomically or durably establish the business effect under an event-level idempotency key; then mark the delivery complete; then acknowledge success. If the process crashes after the effect but before the delivery mark, redelivery finds the business event already applied, performs no second effect, repairs the delivery completion record, and can safely acknowledge.

Real systems may implement this with a database transaction, a unique constraint, an inbox table, an idempotent command record, an outbox/inbox pair, or provider-specific infrastructure. This lesson does not declare one mechanism universally best. It requires an inspectable invariant: one semantic event cannot cause the protected effect twice under the tested redelivery window.

6. Worked example: invoice status

Assume a provider sends a signed payload for invoice inv-42:

{"event_id":"evt-102","type":"invoice.paid","invoice_id":"inv-42","version":2,"amount":1200}

The request also carries a provider signature and delivery ID del-9002. The receiver first verifies the signature against the exact raw string. Only after verification does it parse JSON. It checks whether evt-102 already exists in the applied-event record. If not, it compares version 2 with the stored invoice version and applies the new state. It records evt-102 as applied, marks del-9002 complete, then returns success.

Now inject a crash after the invoice update and event-level idempotency record are durable but before del-9002 is marked complete. On redelivery, authenticity still passes. The delivery record is incomplete, but the event record already contains evt-102. The receiver does not charge or transition the invoice twice; it records the delivery completion and acknowledges.

This example intentionally separates three facts: the message is authentic, the semantic event is already applied, and this delivery attempt is now complete. Keeping those facts separate makes the recovery behavior testable.

7. Failure modes and mutations

Mutation A — verify a reserialized body. Parse the incoming JSON, call JSON.stringify, and verify that new string. A payload containing insignificant whitespace now produces a different byte sequence. Expected evidence: the otherwise valid delivery fails authenticity verification. Repair: verify the original raw body before parsing.

Mutation B — skip authenticity. Accept any body with a plausible event ID. Expected evidence: a forged event changes state. Repair: require provider-defined signature verification before the payload becomes trusted input.

Mutation C — delivery-only dedupe. Record only completed delivery IDs and crash after the business effect but before the delivery mark. Expected evidence: redelivery can repeat the side effect. Repair: add business-event idempotency at the effect boundary.

Mutation D — mark delivery complete before the effect. If the process crashes after marking completion but before applying the effect, redelivery may be suppressed as “already processed.” Expected evidence: acknowledged/completed delivery with missing business effect. Repair: make the effect durable before the success/completion boundary, or use a transaction/state machine that preserves the invariant.

Mutation E — trust arrival order. Apply every invoice version in arrival order. Deliver version 3, then late version 2. Expected evidence: the invoice rolls backward. Repair: compare domain version/sequence and record stale/gap conditions for reconciliation.

8. Experienced bridge: define the acknowledgement contract

“Return quickly” is incomplete advice. A receiver needs an explicit acknowledgement policy tied to its durability model. If the endpoint returns 2xx before any durable handoff, a process failure can lose work while the provider believes delivery succeeded. If it waits for slow downstream work, the provider may time out and redeliver, increasing duplicate pressure.

A common bounded design is to authenticate, validate the minimal envelope, durably enqueue or record the event under an idempotency key, and then acknowledge. Business processing can continue asynchronously. Another design performs a small durable state transition synchronously. Both can be valid when the success condition is explicit and tested.

In the local fixture, success is returned only after the business effect/idempotency state and delivery completion state satisfy the local invariant. This does not prove that a distributed production system has the same failure behavior; it proves the fixture's state transition contract.

Ordering, gaps, and reconciliation

Webhook delivery should not be treated as globally exactly-once or globally ordered by default. A domain event may arrive late, be duplicated, or be missing when another version appears first. The receiver needs domain-aware recovery rather than transport optimism.

For the invoice fixture, each event contains an invoice version. If the current version is 3 and version 2 arrives later, the receiver records it as stale and does not roll state backward. If the current version is 1 and version 3 arrives, the receiver can apply or stage version 3 according to policy while recording a reconciliation requirement because version 2 is missing. A production system might fetch current provider state, query an event log, or schedule reconciliation. The lesson does not prescribe the external mechanism; it requires the gap to remain observable rather than silently disappearing.

This is also where provider delivery IDs and business versions must remain distinct. A new delivery ID can carry a business event already seen. The same business object can also receive multiple legitimate events with different versions.

9. Alternatives and trade-offs

Unique constraint / inbox table: strong and inspectable when a relational store owns the effect boundary. It can make duplicate acceptance cheap, but schema and transaction design matter.

Distributed cache dedupe: fast, but expiry and eviction can reopen the duplicate window. It may be suitable for a bounded risk window, not as an automatic substitute for durable business idempotency.

Queue-first acknowledgement: useful when a durable broker is the reliability boundary. The receiver must still authenticate before trusting/enqueuing and must make downstream consumers idempotent.

Provider reconciliation API: useful for gaps or missed events, but it is provider-specific and may require credentials/network access. The local lab deliberately does not depend on it.

Timestamp freshness checks: can narrow replay windows when the provider signs a timestamp and specifies validation rules, but a timestamp alone is not universal replay prevention. Provider contracts differ; event/delivery identity and business idempotency may still be required.

10. Guided and independent practice

Guided task: run the reference scenario mentally before code. Predict the state after an authentic version-2 event. Predict the state after the exact same delivery is repeated. Then predict the state after a crash occurs after the durable effect but before delivery completion.

Implementation task: repair the learner fixture so the authentic baseline passes, a forged signature is rejected, a duplicate delivery does not repeat the effect, the crash/redelivery path converges to one effect, and stale versions do not roll state backward.

Independent transfer: adapt the contract to a shipping provider sending shipment.updated events. Use different header names and a different payload schema. Preserve the architectural separation: provider authenticity, event/delivery identity, durable effect, acknowledgement policy, ordering/gap reconciliation. Do not copy the GitHub header into a fictional provider contract unless the scenario explicitly defines it.

11. Verification evidence

Your evidence record needs more than “tests pass.” Record the implementation artifact, the exact observation for each contract row, the inference you draw from that observation, at least three mutation results, the repair result, a changed-context transfer result, and one residual limitation.

A strong authenticity row says: “With the exact raw body, the expected HMAC matches and processing proceeds; after reserialization, the same provider signature no longer matches.” The inference is bounded: “This fixture verifies the configured provider-style signature over the raw input.” It does not say: “All webhook providers use this header,” or “HMAC prevents replay.”

A strong crash-recovery row records the effect count before the crash, the failed acknowledgement, the redelivery, and the final effect count. The inference is: “Within this fixture, event-level idempotency prevents a duplicate effect across the injected crash window.” Residual limitation: the in-memory/local persistence model is not evidence about a real distributed datastore, queue, proxy, or provider retry schedule.

12. Assessment and transfer

Retrieval: state the difference between authenticity and replay protection. Application: implement the raw-body verification and duplicate-safe effect boundary. Diagnosis: explain why a crash between effect and delivery completion is dangerous. Decision: compare delivery-ID dedupe with business-event idempotency for the billing scenario. Transfer: redesign the same contract for a different provider whose signature header and event schema differ.

Blocking evidence requires an executable result for authenticity, duplicate/replay behavior, crash recovery and ordering/reconciliation. Visual inspection or a successful 2xx alone is insufficient.

13. References and freshness boundary

The frozen source boundary uses current GitHub webhook validation guidance, GitHub repository-webhook delivery/redelivery API documentation, and RFC 2104 for HMAC. The current recheck on 2026-09-13 found the frozen claims consistent with those official sources. GitHub is used as a bounded provider example; RFC 2104 defines the HMAC construction but does not supply application replay state.

Review the volatile provider guidance no later than 2026-10-17 or earlier if the provider changes signature, delivery, or API behavior. The lesson's architectural separation can remain useful even when a provider-specific header or endpoint changes, but the provider claim must be rechecked rather than assumed.