Skip to content

How to design a webhook delivery system

The first sections are free. The deep dives, level expectations and quiz are part of Pro.

On this page
Asked at
Practice this question

The problem

A webhook delivery system takes each event your platform produces, finds the customer endpoints subscribed to that event type, and sends each one a signed HTTP POST, trying again when the customer's server fails to answer. Customer servers are slow or down a good part of the time, so the answer turns on two ideas: a failed send is routine work that waits in a durable store for its next try, and one slow or broken endpoint has to be kept from holding the delivery workers that all the other customers share.

Clarifying questions

AskWhy it mattersA typical answer

How many events a day, and how spiky is the traffic?

The event rate sizes the delivery worker pool and the retry store, and the peak decides whether you design for the average or for the burst.

Up to 1 billion events a day, about 11,600 a second on average, with peaks 3 to 5 times that, so roughly 35,000 to 58,000 a second.

How long do we keep retrying a failing endpoint, and what happens to an event after that?

The window sets how much pending work the retry store holds during a long outage, and it decides whether a replay path is needed at all.

Retry for up to 24 hours with growing waits between tries, then mark the delivery failed and open it to replay by the customer.

Do customers need events in the order they happened?

Strict order per endpoint changes the delivery model: it means a single outstanding request to that endpoint, which caps throughput and makes a retry block everything behind it.

Not in the base version. Treat it as a follow-up and be ready to say what strict order would cost.

Do we send the whole event body, or an id the customer fetches?

A full body means no second call for the receiver but bigger payloads to store and resend; a thin event is small and makes stale order matter less.

Full bodies by default. Have a view on thin events, and be ready to pick one side and defend it.

How quickly should an event land when the endpoint is healthy?

A few seconds rules out batching deliveries for later and means the retry store is for failures, not for the main path.

Within a few seconds while the endpoint is healthy.

Where do events come from, and do we own that code?

If producers are out of scope, the design starts at an ingest API, and the first job is to make accepting an event cheap and durable so a slow customer can't slow the producer.

Producing events is out of scope. Events arrive at an ingest API you own, and the design starts there.

The requirements menu

RequirementScopeWhy

Customers register an endpoint URL and choose the event types it receives

Core

The subscription table everything else reads. Registering also hands the customer a signing secret, so this call is where trust starts.

Every matching event is sent to the endpoint as a signed HTTP POST

Core

The main path: fan out each event to its subscribed endpoints and send a request the receiver can verify.

Failed deliveries are retried with backoff for a fixed window, then marked failed

Core

Receivers fail often enough that retries are part of normal traffic, and a fixed window bounds how much pending work you hold.

Customers see each event’s attempts and status, and can replay an event

Core

Without it, a customer whose endpoint was down discovers missing orders days later and has no way back. It also forces you to record each attempt, which the retry design needs anyway.

Pause an endpoint after a long run of failures and tell the customer

Optional

Worth taking: it stops workers from spending sends on a dead endpoint and gets the customer a clear signal. Leave it if time is short, since caps per endpoint already contain the damage.

Let customers rotate an endpoint's signing secret

Optional

Real platforms need it, and it tests whether you can keep two secrets valid during the overlap. Mention it and move on unless the interviewer asks about trust.

Deliver one endpoint's events in strict order

Optional

Tempting because customers ask for it. Leave it out of the base design and name the price: a single outstanding request per endpoint and a retry that blocks each later event.

Thin events: send the event id and type, and let the customer fetch the current state

Optional

Smaller payloads and no stale data, at the cost of one more call per event for the customer. Pick a side if asked.

A per-endpoint rate limit the customer can set

Optional

A cap on outstanding requests already bounds the pressure on one endpoint; a customer-set rate is a refinement for receivers with their own limits. Leave it unless asked.

Producing the events inside the platform

Out of scope

The platform's own services decide what counts as an event. You take events from an ingest API and start from there.

The customer’s code that handles each event

Out of scope

You can document what a receiver should do (acknowledge fast, skip ids already handled, verify the signature) but you don't build it.

Streaming APIs that replace webhooks with a long-lived connection

Out of scope

A different product with different failure modes. Name it as an alternative for customers who want order and skip it.

Non-functional requirements

RequirementWhat it forces

No accepted event is dropped silently: at-least-once delivery

The ingest API writes the event to a durable log before it acknowledges the producer, and a pending retry lives in a durable store rather than a worker's memory. It also decides the contract: duplicates are possible, so each request carries a stable event id the receiver can check.

Delivery within a few seconds while the endpoint is healthy

The main path is log, fan-out, send, with no batching or waiting for a timer. Retries are a side path, and healthy endpoints must not sit behind sick ones.

Up to 1 billion events a day, with peaks of 3 to 5 times the average

About 11,600 events a second on average and up to 58,000 at peak. Workers scale horizontally and hold no state, and the pending store takes a write per attempt, so it's indexed by next try time and partitioned rather than a single table that each worker scans.

One slow or failing endpoint never delays other customers

The delivery workers are the shared resource, so each endpoint gets a cap on outstanding requests and a short timeout, the scheduler rotates across endpoints instead of reading in arrival order, and an endpoint that keeps failing is paused while its events wait.

Receivers can check that a request really came from the platform

A secret per endpoint, handed out at registration, and an HMAC over a timestamp and the raw body in a header on each request. Receivers reject signatures that don't match and timestamps that are too old, which blocks a captured request from being replayed later.

Estimates

QuantityWorkingResultWhat it sizes

Requests in flight at peak with healthy endpoints

Assume a healthy send holds a worker for about 300 ms (a few hundred milliseconds is a typical job time in published numbers). 58,000 events a second × 0.3 s ≈ 17,400 requests in flight. If all endpoints answered in ten seconds: 58,000 × 10 s = 580,000.

About 17,400 outstanding requests at peak, or about 580,000 if the whole world slowed to ten seconds

The delivery worker pool, and why response time, not request count, is what a slow endpoint costs you.

Requests one slow endpoint ties up without a cap

Example customer receiving 1 million events an hour: 1,000,000 ÷ 3,600 s ≈ 278 events a second. At ten seconds a response, each outstanding request carries 0.1 events each second, so keeping up needs 278 ÷ 0.1 ≈ 2,780 requests in flight. 17,400 ÷ 2,780 ≈ 6.3.

About 2,780 requests in flight for that one endpoint, about a sixth of the whole peak pool

The per-endpoint cap on outstanding requests: anything below 2,780 makes that customer's backlog grow in their own lane rather than in the shared pool.

Pending retries when one large customer is down for a day

Example customer with 1% of all events: 1% × 1 billion = 10 million events a day. Assume a median payload of about 5 KB: 10 million × 5 KB ≈ 50 GB of bodies waiting, plus one row per attempt.

About 10 million pending deliveries and about 50 GB of payloads for one customer

The retry store: it must hold tens of millions of rows for one endpoint and still answer 'what is due now for endpoint X' quickly.

Attempts in a 24-hour window with doubling waits

Waits of 1, 2, 4, ..., 512 minutes sum to 1,023 minutes ≈ 17 hours, so about eleven attempts fit in 24 hours before the next wait (1,024 minutes) would pass the window. A fixed one-minute interval instead would mean 1,440 sends per event a day.

About eleven attempts per failed event in 24 hours, against 1,440 with a fixed one-minute interval

The number of attempt rows per failed event, and why doubling waits are what make a 24-hour window affordable.

Retry write rate on an ordinary day

Published production figures put events that succeed on a retry at about 1.5%: 1.5% × 1 billion = 15 million events a day. 15 million ÷ 86,400 s ≈ 174 events entering the retry schedule each second on average, before any outage. That's a floor for the store's write rate: each retried event takes at least one write, and several attempts multiply it.

About 15 million retried events a day, about 174 a second

The baseline write load on the retry store; an outage at one large customer adds the 10 million above on top.

Payload storage for retries and replay

1 billion events × 5 KB = 5 TB a day. The 24-hour retry window needs 5 TB hot; keeping events replayable for 15 days needs 15 × 5 TB = 75 TB.

About 5 TB a day, about 75 TB for a fifteen-day replay window

The event store and its retention: bodies belong in cheap object storage or a log with retention, not in the retry table.

Lanes if each customer and endpoint pair gets its own queue

A published production figure: 42,000 active sources × 2.1 endpoints each ≈ 88,000 queues.

About 88,000 queues for one real sender, growing with customers

Why a queue per customer and endpoint is more queues than you'd want to run as fixed topics, and why a database you can query by endpoint replaces fixed queues.

Stalls a year with one shared queue

The published estimate: 200+ endpoints, each unavailable about an hour a year (99.9% available). 200 × 1 hour ≈ 200 stalled hours a year, which is a stall most days.

Roughly one hour-long stall a day with a single shared queue

Why isolation per endpoint is a core requirement rather than a refinement.

Core entities

EntityWhat it holds

Endpoint

One customer URL and what it wants: event types, the secret used to sign requests to it, and whether it's active or paused.

Event

One thing that happened, written once to the log and kept long enough to retry and replay it.

Delivery

One event bound for one endpoint: its current state and when it's next due. This is the row the retry scheduler reads.

Delivery attempt

One request actually sent, with what came back. Append-only, and what the customer sees on the deliveries page.

The API

EndpointNotes

POST /endpoints { url, event_types } → { endpoint_id, signing_secret }

Checks that the URL is https and the event types exist, then generates the secret and returns it. In this design the secret is handed over at registration for the customer to store, and changing it later goes through a separate rotation flow. Consider a test ping to the URL before activating it so a typo surfaces now rather than as a day of failed attempts.

GET /events/{id}/deliveries → attempts with status codes and times

Reads the attempt rows for the caller's own event. Each attempt shows its number, status code or error, latency, time, and the delivery's next scheduled try, so a customer debugging their endpoint sees the same picture your workers see.

POST /events/{id}/replay { endpoint_id }

Inserts a new due row for that event and endpoint, so a replay goes through the same scheduler and caps as a first attempt. The request carries the same event id, so a receiver that already handled it can skip it. Allowed for events still in retention, and idempotent per event and endpoint while a replay is pending.

Outbound: POST {endpoint url} with the event body, its event id and a signature header

The body is the event JSON. Headers carry the event id, the delivery id, the attempt number, a timestamp and an HMAC of the timestamp plus the raw body under the endpoint's secret. Any 2xx within the timeout counts as delivered; anything else is classified before a retry is scheduled.

The high-level design

appendsigned POSTIngest APIEvent logDelivery workersCustomerendpoint

Step 1 of 4

Write the event down, then send it

Start with the smallest shape that keeps the producer safe. An ingest API takes the event, appends it to a durable log and acknowledges the producer at once; the producer doesn't wait on a customer's server. Delivery workers read the log and send each event as a POST to the customer's URL. This already avoids a common mistake, sending the request inline from the producer. What it leaves open: the workers have to work out which endpoints want each event, and if a worker handles one event's several endpoints as a unit, a failure at one of them can delay the others.

appendlookupsigned POSTrecordduereplayIngest APIEvent logFan-out workersEndpoints DBPer-endpointcapsDelivery workersCustomerendpointDeliveries, nexttryCustomer API
The whole design

What breaks first

The deep dives

Retries and a failing endpoint. A customer's endpoint starts returning errors and stays down for six hours while their events keep arriving. What happens to each event, where does it wait between tries, and what does the customer see when the window closes?

A concept that helps here: Retries and dead-letter queuesA concept that helps here: Job scheduling and delayed workA concept that helps here: Backpressure and load shedding

One customer slowing everyone down. One large customer's endpoint takes ten seconds to answer while millions of their events wait. What in your design runs out first, and how do the other customers' events keep landing within seconds?

A concept that helps here: Multi-tenant isolationA concept that helps here: Backpressure and load sheddingA concept that helps here: Rate limiting algorithms

Duplicates, order and trust. A customer's server got the same payment event twice and shipped two orders. Where did the second copy come from, what must each request carry so the receiver can drop it, and how does the receiver know a request is really yours and not an old one sent again?

A concept that helps here: Delivery semantics: at most, at least, exactly onceA concept that helps here: Idempotency keysA concept that helps here: Clocks and ordering

Concepts this guide uses

Sources

  1. Segment: Centrifuge, a reliable system for delivering billions of events per day
  2. Stripe docs: Receive Stripe events in your webhook endpoint
  3. Shopify docs: Verify webhook deliveries
  4. Hookdeck: Shopify webhooks in production, duplicates, ordering, retries and timeouts
  5. Marc Brooker: Jitter, making things better with randomness
Sign in to report a mistake