How to design an ad click aggregator
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Counting each click once
- Scaling writes and hot ads
- Losing no clicks
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
An ad click aggregator sits between a click on an ad and the advertiser's landing page: it records the click, sends the browser on with a redirect, and turns the flow of clicks into per-ad, per-minute counts that advertisers read and are billed on. Each click is money, so the design turns on two pressures: retried and forged clicks should stay out of the counts, and a peak of 10,000 clicks a second can land mostly on one celebrity ad, which pins a single stream shard. The shape that handles both is a click processor that checks a signed impression id and writes to a stream partitioned by ad, a stream job that produces minute counts on event time, a separate store for advertiser queries, and raw clicks kept long enough that a nightly recount can correct the live counts.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
How many clicks a day, how many ads, and how spiky is the peak? | The peak rate sizes the click processors and the number of stream shards, and the share of clicks on the busiest ad tells you whether one hot ad can pin a shard. | About 10 million active ads and about 100 million clicks a day, which is about 1,000 clicks a second on average and 10,000 at peak. A bigger variant of the question assumes 1 billion clicks a day across 2 million ads. |
How fresh do the counts have to be, and at what granularity? | Minute granularity with counts visible within minutes rules out a nightly batch as the main path and points at stream counting with small windows. | Advertisers see counts per minute, visible within a few minutes of the click and as close to real time as you can get. |
Does the click have to pass through our servers, or can the browser go straight to the advertiser and report the click separately? | A server-side redirect sees each click at the cost of one more hop; a client-side report is simpler but a user or a blocker can skip it, so the counts drift from reality. | Route the click through us: the browser posts the click and we answer with a redirect to the advertiser's URL. |
How exact do the counts have to be? Is this billing, or a dashboard where a small error is fine? | Billing changes the design: a lost click is lost revenue and a double count is an overcharge, so you need an impression-level duplicate check and a way to correct counts after a crash. | Advertisers are billed on these counts, so no click may be lost and none counted twice. A live count that is corrected a little later by a recount is acceptable. |
Which filters do advertisers need: country, device, placement? How many dimensions? | Each dimension multiplies the number of pre-aggregated rows, and many dimensions push you toward a columnar store that filters at query time rather than a time-series table. | A small set such as country. Design so adding a dimension is a schema change, not a redesign. |
Is fraud detection part of this, or do we just need to reject clicks we can prove are forged or replayed? | Full fraud detection is a product of its own; rejecting a click with a bad signature or a reused impression id is cheap and belongs in the click path. | Fraud and spam detection are out of scope. Do reject a click whose signature fails or whose impression id the processor met before. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
A peak of 10,000 clicks a second | The click processors must scale out behind a load balancer and do little work per click: a signature check, a cache lookup and one write to a stream. The stream needs enough shards for the peak by record count, and one popular ad must not pin a single shard. |
Advertiser queries answer in under a second | Queries read pre-aggregated minute rows from a store built for analytical reads, kept apart from the click writes. Long ranges read daily and weekly rollups rather than tens of thousands of minute rows. |
No click lost, and none counted twice | A click is written to a replicated stream before the browser gets its redirect, and the duplicate check uses a per-impression id rather than a user id. Raw clicks are kept so a recount can find and fix a count that the live path got wrong. |
Counts visible within a few minutes end to end, as close to real time as possible | Counting happens in a stream job with one-minute windows on event time that flushes often, not in a batch that runs each few minutes. The watermark that waits for late clicks is kept short, and whatever it misses is fixed by the recount. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Average and peak click rate | 100 million clicks a day ÷ 86,400 seconds ≈ 1,157 a second, call it 1,000; the peak is taken as 10 times the average, 10,000 a second | About 1,000 clicks a second on average, 10,000 at peak | The click processors and the stream: design for the peak, since a celebrity ad or a sale makes it real. |
Stream shards for the peak | By records: 10,000 clicks a second ÷ a shard's write limit of 1,000 records per second = 10 shards. By bytes: 10,000 × 100 bytes = 1 MB a second, which is about one shard's byte limit | At least 10 shards, and the record limit binds before the byte limit | How many stream shards the peak needs, with headroom on top, since partitioning by ad id spreads clicks unevenly. |
Suffixes for one hot ad | If one ad draws most of the 10,000 clicks a second and a shard's write limit is 1,000 records per second, its clicks need to land on at least 10,000 ÷ 1,000 = 10 shards. Partition keys are hashed to shards, so distinct suffixes can collide on a shard; doubling to 20 keys gives headroom for that | About 10 to 20 suffixed keys for the hottest ad | How many partial counts the stream job adds back together for a popular ad. |
Memory for one day of impression ids | 100 million clicks a day × 16 bytes per id = 1.6 GB; ids expire after a day so the set does not grow past that | About 1.6 GB, plus Redis overhead per key | The duplicate-check cache: a small Redis cluster with a replica, not a database. |
Ids at risk when the dedup cache crashes | The append log synced once a second loses about one second of writes: 1 × 10,000 = up to 10,000 ids at peak. A snapshot taken once in five minutes loses up to 5 × 60 × 10,000 = 3 million ids | Up to 10,000 ids with per-second log sync, up to 3 million with snapshots and no log | Why the cache runs with the append log and a replica, and why a duplicate that slips through is left to the recount. |
Raw click storage | 100 million clicks a day × 100 bytes = 10 GB a day. Kafka's default retention is 168 hours (7 days), so the stream holds about 70 GB; a year in the object store is about 3.65 TB | About 10 GB a day, 70 GB on the stream, 3.65 TB a year archived | The disks behind the stream and the object store that the recount reads. Both are small enough that keeping raw clicks is an easy call. |
Minute rows a long-range query would read | 24 × 60 = 1,440 minutes a day; a 30-day range is 30 × 1,440 = 43,200 minute rows per ad per filter | 43,200 rows for one ad over 30 days, before rollups | Why the design keeps daily and weekly rollups: a month-long query reads 30 daily rows instead. |
Size of one batch in the batch alternative | 10,000 clicks a second × 300 seconds (a 5-minute interval) × 100 bytes = 300 MB per batch at peak | About 300 MB per 5-minute batch | The batch job's input at peak. Small enough that batch is feasible; the objection to it is lag, not volume. |
Core entities
| Entity | What it holds |
|---|---|
Click event | One raw click as it reached the processor. Written to the stream and archived; the source of truth that recounts read. |
Aggregated count | One row per ad, per minute, per filter value. What advertiser queries read; rebuilt from raw clicks when a recount disagrees. |
Ad | The thing being clicked: who pays for it and where the redirect goes. |
Impression | One showing of one ad to one browser. The processor does not store it; it checks a signature the placement service put on it. |
The API
| Endpoint | Notes |
|---|---|
| The hot path: recompute the signature, reject a mismatch, look the impression id up in the seen-id cache, append the click record to the stream, mark the id seen, then answer with the redirect. A bad signature or a seen id still redirects, so the user lands where they expected; it just isn't counted. A |
| Served from the analytical store. Short ranges sum minute rows; ranges over a day or more read the daily and weekly rollups and add minute rows for the partial days at the edges. The caller must own the ad. |
| The bonus query. Answered from a small table the stream job rewrites each minute with the top ads over a sliding window of the last few minutes, rather than ranking 10 million ads at request time. |
The high-level design
Step 1 of 4
Record the click, then redirect
Start with the straight line. The browser posts the click to a click processor, which writes a row to a counts database and answers with a 302 to the advertiser's URL. The advertiser dashboard reads the same database. This passes the functional requirements and gives you something to point at. Its problems are the ones the rest of the design fixes: at 10,000 clicks a second the writes and the dashboard's range sums fight over one database, and a processor that dies after answering the redirect but before the write finishes has lost a click with no trace.
What breaks first
The deep dives
Counting each click once. A browser retries a click and the same click reaches the processor twice; a bad actor replays or forges one. How does the processor count the click once, and what happens if it crashes partway through?
Scaling writes and hot ads. A celebrity ad goes live and draws most of the 10,000 clicks a second. Which component gives out first, how does one of those clicks reach its minute count, and how do you keep counts near real time?
Losing no clicks. The counting job crashes in the middle of a minute, and days later some advertisers' counts turn out wrong. How do you notice, how do you fix the stored counts, and how do you keep the fix from double counting?
A concept that helps here: Lambda and Kappa architectures
A concept that helps here: Queues and logsConcepts this guide uses
Sources
- Alex Xu, System Design Interview volume 2, chapter 22: Ad click event aggregation (book notes)
- Amazon Kinesis Data Streams documentation: Create and manage Kinesis data streams
- Amazon Kinesis Data Streams FAQs
- Confluent: Apache Kafka retention explained
- Redis documentation: Redis persistence
- Apache Flink documentation: Streaming analytics, event time and watermarks
- Apache Flink documentation: Generating watermarks