How to design a webhook delivery system
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Retries and a failing endpoint
- One customer slowing everyone down
- Duplicates, order and trust
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
A webhook delivery system takes each event your platform produces, finds the customer endpoints subscribed to that event type, and sends each one a signed HTTP POST, trying again when the customer's server fails to answer. Customer servers are slow or down a good part of the time, so the answer turns on two ideas: a failed send is routine work that waits in a durable store for its next try, and one slow or broken endpoint has to be kept from holding the delivery workers that all the other customers share.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
How many events a day, and how spiky is the traffic? | The event rate sizes the delivery worker pool and the retry store, and the peak decides whether you design for the average or for the burst. | Up to 1 billion events a day, about 11,600 a second on average, with peaks 3 to 5 times that, so roughly 35,000 to 58,000 a second. |
How long do we keep retrying a failing endpoint, and what happens to an event after that? | The window sets how much pending work the retry store holds during a long outage, and it decides whether a replay path is needed at all. | Retry for up to 24 hours with growing waits between tries, then mark the delivery failed and open it to replay by the customer. |
Do customers need events in the order they happened? | Strict order per endpoint changes the delivery model: it means a single outstanding request to that endpoint, which caps throughput and makes a retry block everything behind it. | Not in the base version. Treat it as a follow-up and be ready to say what strict order would cost. |
Do we send the whole event body, or an id the customer fetches? | A full body means no second call for the receiver but bigger payloads to store and resend; a thin event is small and makes stale order matter less. | Full bodies by default. Have a view on thin events, and be ready to pick one side and defend it. |
How quickly should an event land when the endpoint is healthy? | A few seconds rules out batching deliveries for later and means the retry store is for failures, not for the main path. | Within a few seconds while the endpoint is healthy. |
Where do events come from, and do we own that code? | If producers are out of scope, the design starts at an ingest API, and the first job is to make accepting an event cheap and durable so a slow customer can't slow the producer. | Producing events is out of scope. Events arrive at an ingest API you own, and the design starts there. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
No accepted event is dropped silently: at-least-once delivery | The ingest API writes the event to a durable log before it acknowledges the producer, and a pending retry lives in a durable store rather than a worker's memory. It also decides the contract: duplicates are possible, so each request carries a stable event id the receiver can check. |
Delivery within a few seconds while the endpoint is healthy | The main path is log, fan-out, send, with no batching or waiting for a timer. Retries are a side path, and healthy endpoints must not sit behind sick ones. |
Up to 1 billion events a day, with peaks of 3 to 5 times the average | About 11,600 events a second on average and up to 58,000 at peak. Workers scale horizontally and hold no state, and the pending store takes a write per attempt, so it's indexed by next try time and partitioned rather than a single table that each worker scans. |
One slow or failing endpoint never delays other customers | The delivery workers are the shared resource, so each endpoint gets a cap on outstanding requests and a short timeout, the scheduler rotates across endpoints instead of reading in arrival order, and an endpoint that keeps failing is paused while its events wait. |
Receivers can check that a request really came from the platform | A secret per endpoint, handed out at registration, and an HMAC over a timestamp and the raw body in a header on each request. Receivers reject signatures that don't match and timestamps that are too old, which blocks a captured request from being replayed later. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Requests in flight at peak with healthy endpoints | Assume a healthy send holds a worker for about 300 ms (a few hundred milliseconds is a typical job time in published numbers). 58,000 events a second × 0.3 s ≈ 17,400 requests in flight. If all endpoints answered in ten seconds: 58,000 × 10 s = 580,000. | About 17,400 outstanding requests at peak, or about 580,000 if the whole world slowed to ten seconds | The delivery worker pool, and why response time, not request count, is what a slow endpoint costs you. |
Requests one slow endpoint ties up without a cap | Example customer receiving 1 million events an hour: 1,000,000 ÷ 3,600 s ≈ 278 events a second. At ten seconds a response, each outstanding request carries 0.1 events each second, so keeping up needs 278 ÷ 0.1 ≈ 2,780 requests in flight. 17,400 ÷ 2,780 ≈ 6.3. | About 2,780 requests in flight for that one endpoint, about a sixth of the whole peak pool | The per-endpoint cap on outstanding requests: anything below 2,780 makes that customer's backlog grow in their own lane rather than in the shared pool. |
Pending retries when one large customer is down for a day | Example customer with 1% of all events: 1% × 1 billion = 10 million events a day. Assume a median payload of about 5 KB: 10 million × 5 KB ≈ 50 GB of bodies waiting, plus one row per attempt. | About 10 million pending deliveries and about 50 GB of payloads for one customer | The retry store: it must hold tens of millions of rows for one endpoint and still answer 'what is due now for endpoint X' quickly. |
Attempts in a 24-hour window with doubling waits | Waits of 1, 2, 4, ..., 512 minutes sum to 1,023 minutes ≈ 17 hours, so about eleven attempts fit in 24 hours before the next wait (1,024 minutes) would pass the window. A fixed one-minute interval instead would mean 1,440 sends per event a day. | About eleven attempts per failed event in 24 hours, against 1,440 with a fixed one-minute interval | The number of attempt rows per failed event, and why doubling waits are what make a 24-hour window affordable. |
Retry write rate on an ordinary day | Published production figures put events that succeed on a retry at about 1.5%: 1.5% × 1 billion = 15 million events a day. 15 million ÷ 86,400 s ≈ 174 events entering the retry schedule each second on average, before any outage. That's a floor for the store's write rate: each retried event takes at least one write, and several attempts multiply it. | About 15 million retried events a day, about 174 a second | The baseline write load on the retry store; an outage at one large customer adds the 10 million above on top. |
Payload storage for retries and replay | 1 billion events × 5 KB = 5 TB a day. The 24-hour retry window needs 5 TB hot; keeping events replayable for 15 days needs 15 × 5 TB = 75 TB. | About 5 TB a day, about 75 TB for a fifteen-day replay window | The event store and its retention: bodies belong in cheap object storage or a log with retention, not in the retry table. |
Lanes if each customer and endpoint pair gets its own queue | A published production figure: 42,000 active sources × 2.1 endpoints each ≈ 88,000 queues. | About 88,000 queues for one real sender, growing with customers | Why a queue per customer and endpoint is more queues than you'd want to run as fixed topics, and why a database you can query by endpoint replaces fixed queues. |
Stalls a year with one shared queue | The published estimate: 200+ endpoints, each unavailable about an hour a year (99.9% available). 200 × 1 hour ≈ 200 stalled hours a year, which is a stall most days. | Roughly one hour-long stall a day with a single shared queue | Why isolation per endpoint is a core requirement rather than a refinement. |
Core entities
| Entity | What it holds |
|---|---|
Endpoint | One customer URL and what it wants: event types, the secret used to sign requests to it, and whether it's active or paused. |
Event | One thing that happened, written once to the log and kept long enough to retry and replay it. |
Delivery | One event bound for one endpoint: its current state and when it's next due. This is the row the retry scheduler reads. |
Delivery attempt | One request actually sent, with what came back. Append-only, and what the customer sees on the deliveries page. |
The API
| Endpoint | Notes |
|---|---|
| Checks that the URL is |
| Reads the attempt rows for the caller's own event. Each attempt shows its number, status code or error, latency, time, and the delivery's next scheduled try, so a customer debugging their endpoint sees the same picture your workers see. |
| Inserts a new due row for that event and endpoint, so a replay goes through the same scheduler and caps as a first attempt. The request carries the same event id, so a receiver that already handled it can skip it. Allowed for events still in retention, and idempotent per event and endpoint while a replay is pending. |
| The body is the event JSON. Headers carry the event id, the delivery id, the attempt number, a timestamp and an HMAC of the timestamp plus the raw body under the endpoint's secret. Any |
The high-level design
Step 1 of 4
Write the event down, then send it
Start with the smallest shape that keeps the producer safe. An ingest API takes the event, appends it to a durable log and acknowledges the producer at once; the producer doesn't wait on a customer's server. Delivery workers read the log and send each event as a POST to the customer's URL. This already avoids a common mistake, sending the request inline from the producer. What it leaves open: the workers have to work out which endpoints want each event, and if a worker handles one event's several endpoints as a unit, a failure at one of them can delay the others.
What breaks first
The deep dives
Retries and a failing endpoint. A customer's endpoint starts returning errors and stays down for six hours while their events keep arriving. What happens to each event, where does it wait between tries, and what does the customer see when the window closes?
One customer slowing everyone down. One large customer's endpoint takes ten seconds to answer while millions of their events wait. What in your design runs out first, and how do the other customers' events keep landing within seconds?
Duplicates, order and trust. A customer's server got the same payment event twice and shipped two orders. Where did the second copy come from, what must each request carry so the receiver can drop it, and how does the receiver know a request is really yours and not an old one sent again?
Concepts this guide uses
Sources
- Segment: Centrifuge, a reliable system for delivering billions of events per day
- Stripe docs: Receive Stripe events in your webhook endpoint
- Shopify docs: Verify webhook deliveries
- Hookdeck: Shopify webhooks in production, duplicates, ordering, retries and timeouts
- Marc Brooker: Jitter, making things better with randomness