How to design a notification system
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Never losing a notification
- Opt-outs, rate limits and priority
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
A notification system is the one service other teams call when they need to reach a user by push, text message or email. It takes a request from a calling service, checks what the user has allowed, fills in a template, and hands the message to a provider it does not run: APNs or FCM for push, a carrier gateway for SMS, a mail provider for email. The answer turns on that dependence. The volume is modest, about 185 sends a second on average, so throughput is not the hard part. The hard parts are keeping the other channels sending while one provider stalls, and making sure a timed-out send is recorded and retried rather than silently dropped.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
Who sends notifications, and how many a day on each channel? | It tells you whether the callers are internal services you can authenticate with a key, and whether throughput or isolation is the hard part. The split by channel also decides how many worker pools you draw. | Internal services and scheduled jobs call the API. About 10 million push notifications, a million text messages and 5 million emails a day, which is about 185 sends a second on average. |
Is a late notification acceptable, and is a lost one? | The answer sets the delivery semantics. If a lost notification is worse than a repeated one, you design for at-least-once delivery with retries and accept rare duplicates. | Soft real time: fast normally, delays are fine under load. Losing a notification is not acceptable. |
Do users control which channels reach them? | A per-channel opt-out means a settings table that sits on the send path, and a decision about where it is checked and how it is cached. | Yes. A user can opt out of each channel separately, and the system has to respect that before anything goes out. |
Which devices and platforms do we support? | Each platform can mean another provider with its own token format, limits and error codes, and a user may have several devices. | iOS, Android and desktop. A user can own more than one device, and each device registers its own token. |
Do urgent notifications, such as a password reset code, share the path with marketing sends? | If they do, a marketing blast can sit in front of a code the user is waiting for right now. Knowing early lets you add a priority lane and a transactional class that skips the per-user cap. | Both go through this service. Treat reset codes and other transactional sends as urgent and keep them from waiting behind bulk sends. |
Does the service need to know whether a message was delivered, or just that the provider accepted it? | Provider acceptance is what the send path can observe directly; delivery status arrives later through callbacks or not at all, and tracking it means a status column that changes over time. | Record acceptance and failure per attempt in a log. Delivery callbacks and open or click tracking are a later addition for tuning volume. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
Soft real time: fast normally, delays are fine under heavy load | The API answers as soon as the message is durably queued and does not wait for a provider. Queues absorb bursts instead of the servers, and the alarm to set is on the age of the oldest queued message, not on per-send latency. |
No notification is lost | A durable queue that stores a message before acknowledging it, a log row per notification, retries with backoff, and a dead-letter queue that alerts a person. Delivery is at-least-once, with the event id used to catch the repeats this system itself creates, because at-most-once gives up on a timed-out send, and that send may be lost. |
10 million push, 1 million SMS and 5 million emails a day | About 185 sends a second on average, with push at about 116 of them. That is a small load: one database with a replica holds the log and the settings, and the design effort goes to isolation and retries rather than sharding. Worker pools are sized per channel from this split. |
An outage at one provider does not stop the other channels | A queue and a worker pool for each provider, short timeouts on provider calls, and no shared pool of threads that a stalled provider can fill. Monitoring is per queue, so an SMS backlog shows up as an SMS problem. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Average sends a second, by channel | 10 million push + 1 million SMS + 5 million email = 16 million a day. Divided by 86,400 seconds: push ≈ 116 a second, SMS ≈ 12 a second, email ≈ 58 a second, about 185 in total | About 185 sends a second: 116 push, 12 SMS, 58 email | The worker pool for each channel, and the observation that the load is small enough that isolation and reliability, not throughput, decide the design. |
Concurrent provider calls each channel needs | As an example assumption (not a provider figure), say a provider call takes about 200 ms, so one worker thread completes about 5 sends a second. Push: 116 ÷ 5 ≈ 23 threads; email: 58 ÷ 5 ≈ 12; SMS: 12 ÷ 5 ≈ 3 | About 23 concurrent push sends, 12 email and 3 SMS at the average rate: a few worker processes per channel | How many workers each queue needs to stay short, and how much headroom to add before a backlog forms. |
SMS backlog after a one-hour provider outage | 12 a second × 3,600 seconds ≈ 43,200 texts waiting. The usual SMS pool is about 3 threads at 5 sends a second each; tripled to 9 threads it sends about 45 a second once the provider returns. New texts keep arriving at 12 a second, so the net drain is 45 - 12 = 33 a second, and 43,200 ÷ 33 ≈ 1,309 seconds, about 22 minutes | About 43,200 queued texts, cleared in about 22 minutes with three times the usual workers | Queue retention, the worker headroom to add after an outage, and how long a reset code would wait unless it carries an expiry. |
Notification log growth | Assume about 200 bytes a row (event id, user, channel, template, status, attempts, timestamps). 16 million rows a day × 200 bytes ≈ 3.2 GB a day; over 30 days ≈ 96 GB | About 3.2 GB a day, about 96 GB a month | Why one database with a replica holds the log, and why a store sharded across regions is more than about 185 sends a second needs. |
A marketing blast compressed into one hour | If a campaign sends the whole day's 5 million emails in one hour: 5,000,000 ÷ 3,600 ≈ 1,389 a second, against the 58 a second average, about 24 times it | About 1,389 emails a second for an hour, about 24 times the average | The email queue depth during a blast, and why a reset code needs its own lane instead of a place in that queue. |
Core entities
| Entity | What it holds |
|---|---|
User | The person being reached, with the addresses the channels need. |
Device | One row per installed app that can take a push; a user can have more than one device. |
Notification setting | Whether a user accepts a channel; read before anything is queued, and cached. |
Template | The wording of a notification with placeholders, filled in from the caller's params. |
Notification log | One row per message per recipient and channel, written before the send and updated after it. |
The API
| Endpoint | Notes |
|---|---|
| In this design each calling service holds a key and secret pair and unsigned requests are rejected, which keeps sends to verified internal callers. The server validates the template and params, expands user_ids into one message per recipient, and checks opt-in. We'd have it answer |
| Writes the settings row, then removes the cached entry for that user and channel so the next send reads the new value. Workers re-check opt-in before sending anything that waited in a backlog, so an opt-out also covers messages already queued. |
| Called by the app when it installs or when the platform hands it a new token. Upserts the device row for the signed-in user. When APNs answers HTTP 410 for a token, the push worker marks that row disabled so later sends skip it. |
The high-level design
Step 1 of 4
Servers that call the providers directly
Start with the simplest shape. Calling services post to notification servers behind a load balancer. The servers check the caller's key, read the user's contact details, devices and opt-in settings from the database, fill in the template, and call the provider for the channel: APNs or FCM for push, the SMS gateway, the mail provider. This already covers the four functional requirements. Its weakness is the provider call inside the request: the caller waits for an outside service, and a slow SMS gateway holds server threads that push and email requests also need.
What breaks first
The deep dives
Never losing a notification. The SMS gateway stops answering while a password reset code is on its way out. Walk one notification from the calling service to the phone, say what happens to the code and to the push and email channels, and say whether a retry can deliver it twice.
Opt-outs, rate limits and priority. A buggy calling service starts sending one user several notifications a minute on all three channels. Where is the opt-out checked, what stops the flood from reaching the phone, and how does a password reset code get through the same protection?
Concepts this guide uses
Sources
- Alex Xu, System Design Interview, chapter 11: Design a notification system (reading notes at books.dwf.dev)
- Apple: Local and Remote Notification Programming Guide, Communicating with APNs
- Firebase Cloud Messaging: Non-collapsible and collapsible messages
- Firebase Cloud Messaging: Understanding message delivery
- The Firebase Blog: Life of a message from FCM to the device
- Amazon SQS Developer Guide: Amazon SQS standard queues
- Amazon SQS Developer Guide: Amazon SQS queue types
- Amazon SQS Developer Guide: Using dead-letter queues in Amazon SQS
- Marc Brooker: Jitter: Making Things Better With Randomness
- OneSignal REST API reference: SMS (idempotency_key)
- Twilio Segment: Delivering billions of messages exactly once
- Microsoft Learn: Rate limiting middleware in ASP.NET Core (token bucket limiter)