How to design a price drop tracker
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Tracking 500 million products
- Trusting prices from the extension
- Fanning out price drop alerts
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
A price drop tracker watches the prices of hundreds of millions of retail products, draws each one's history as a chart, and emails people when a product falls under the number they chose. Two facts shape the answer. There are far more products than you can afford to check often, so a scheduler has to decide which products deserve a fresh look and which can wait. And the second source of prices, a browser extension running in a million strangers' browsers, is free and current but easy to lie through, so its reports need checking before they reach the chart or an inbox. Alerts have an hour to land, which leaves room for a queue and batches rather than a real-time path.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
How many products do we track, and how fresh does each price need to be? | The catalog size against the freshness target decides whether one crawl loop is enough or whether a scheduler has to ration checks across products. | About 500 million products. Prices may be briefly stale, and an alert must go out within 1 hour of the price changing, so products with subscribers need to be checked often enough for that. |
Where do prices come from: an official API, our own crawlers, a browser extension, or all of them? | Each source has a different cost and a different trust level. A crawler fleet is under our control but has a budget the retailer can squeeze; extension reports are free but come from strangers. | Both a crawler fleet and the browser extension. The crawl budget covers a fraction of the catalog at a time, so the extension matters as a second source. |
How many people run the extension, and do we trust what it sends? | A million clients that can post prices is a fan-in problem and an abuse problem at the same time. Knowing the count sizes the ingest path; knowing the trust answer decides whether reports need confirmation. | About 1 million active extension users. Treat what they send as unverified input: nothing from the extension should change the chart or fire an alert until something else agrees with it. |
Does the chart need each recorded price, or is a daily view fine for long ranges? | It decides whether the read path serves raw points or precomputed daily summaries, which is the difference between a cheap query and a slow one on products with years of history. | History queries must answer in under 500 ms. A daily summary for long ranges and raw points for recent ranges is acceptable. |
How bad is a late alert, a missed alert, or a duplicate? | The answer sets the delivery semantics. If a duplicate is merely annoying and a miss is a broken promise, you build for at-least-once delivery and add a record that suppresses repeats. | Alerts should arrive within the hour. A duplicate is annoying but tolerable; a missed alert is the failure people notice. Prices may be briefly stale: availability wins over consistency. |
One retailer or many? | One retailer keeps the product identity simple (one id per product) and keeps the crawler fleet one kind of client. Several retailers means a product matching problem that is a design of its own. | Just Amazon. Comparing prices across retailers is out of scope. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
Handle 500 million products | A product table and a schedule of that size, and a crawl budget that is a small fraction of the catalog per hour. The scheduler has to rank products rather than loop over them, and the price store has to grow with price changes rather than with checks. |
Price history queries answer in under 500 ms | The chart reads one product's points in time order, so the price store is keyed by product and time, long ranges come from daily summaries computed ahead of time, and popular products' charts sit in a cache. |
Alerts go out within 1 hour of a price change | Detection and delivery are decoupled by a queue so a long subscriber list does not stall the crawler, and the matcher finds matching subscriptions by index rather than by scan. The budget is generous enough that batching, not a push connection, is the right tool. |
Availability over consistency: prices may be briefly stale | The read path can serve a cached chart and a slightly old current price. Writes to the price store and alert delivery can be asynchronous and retried, which means the alert path must tolerate at-least-once delivery. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Checks per second if each product is checked on one shared interval | Once a day: 500,000,000 products ÷ 86,400 seconds ≈ 5,787 checks a second. Once an hour: 500,000,000 ÷ 3,600 ≈ 138,900 checks a second | About 5,787 a second for a daily sweep; about 138,900 a second for an hourly one | The crawler fleet under a fixed interval, and why an hourly sweep of the whole catalog is out of reach of any polite crawl budget. |
Checks per second when the interval follows demand (an example split, not a target) | Say 1% of products are hot (have subscribers or traffic): 5,000,000 checked hourly = 5,000,000 ÷ 3,600 ≈ 1,389 a second. The other 495,000,000 checked weekly = 495,000,000 ÷ 604,800 ≈ 818 a second. Total ≈ 2,200 a second | About 2,200 checks a second, under half the daily sweep, while the hot 1% is checked 24 times as often as under a daily sweep | The fleet when checks follow demand, and the argument for a scheduler: freshness where it matters for less than a flat daily sweep. |
Extension reports per second | 1,000,000 users × 20 product pages a day (an assumption) = 20,000,000 reports a day ÷ 86,400 ≈ 231 a second | About 231 reports a second on average | The report ingest path and the per-user rate limit: a free source worth about a tenth of the demand-driven crawl rate, concentrated on popular products. |
Confirmation crawls triggered by reports | If 1 in 100 reports looks like a big drop: 231 ÷ 100 ≈ 2.3 confirmation crawls a second. A flood of 10,000 fake reports for one product in a minute collapses to 1 crawl when confirmations are deduplicated per product | A few crawls a second in normal use; 1 per product under a flood | The cost of confirming reports with a crawl, and why confirmations must be coalesced per product. |
Price store growth: write on change against write on check | Assume a point of product id, price and time takes about 20 bytes of raw payload, before indexes and database overhead (an assumption). Write on change, assuming 50 changes per product a year: 500,000,000 × 50 × 20 bytes ≈ 500 GB a year. Write each daily check: 500,000,000 × 365 × 20 bytes ≈ 3.65 TB a year | About 500 GB a year against about 3.65 TB a year | Why the detector writes a point when the price differs from the last one: storage grows with how often prices move, and the chart loses nothing since a flat line needs no extra points. |
Send rate for a product with a huge subscriber list | Assume 100,000 subscribers on one product (a tenth of the extension's users, an assumption). 100,000 alerts within 3,600 seconds ≈ 28 sends a second. In batches of 100 that is 1,000 batches | About 28 sends a second, or 1,000 batches of 100, spread over the hour | The notification workers and the send queue: the hour budget is generous, so batching is enough and a real-time path is not needed. |
Core entities
| Entity | What it holds |
|---|---|
Product | One row per tracked retailer product: its current price, when it was last checked, and how urgently it should be checked again. |
Price point | A confirmed price for a product at a moment, written when the price differs from the previous one. |
Daily rollup | One row per product per day with low, high and close, so long chart ranges read a few hundred rows instead of each point. |
Subscription | A user's request to hear when a product falls to or under a threshold. |
Price report | An unconfirmed price seen by the extension during a real visit, held until a crawl or enough other reporters agree. |
Sent alert | A record that one subscription was alerted for one price point, so a retried batch does not send the same alert twice; the subscription's armed flag handles bouncing prices. |
User | Whoever subscribes or runs the extension. Accounts are ordinary; the field that matters here is the trust score. |
The API
| Endpoint | Notes |
|---|---|
| Open to anyone in this design, since the chart is the product's front door. Short periods read raw points by product and time; long periods read the daily rollup rows instead, which is how the under 500 ms target holds for a product with years of history. Popular products' responses sit in a cache keyed by product and period and are refreshed when the detector confirms a new point. Pending extension reports are not in this response. |
| In this design the caller must be signed in, and the user id comes from the session, not the body. Writes the subscription row, bumps the product's subscriber count (which raises its crawl priority), and if the product is unknown creates it and schedules a first check. The threshold is stored as integer cents. If the current confirmed price is already at or under the threshold, say so in the response rather than sending an alert. |
| Restricted to the owner, checked against the session. Deletes the row and lowers the product's subscriber count. A batch already on the send queue may still reach this user once; the worker re-reads the subscription before sending and skips a deleted one, which closes most of that window. |
| The reporter id comes from the extension's session, which is what the per-user rate limit and the trust score key on. |
The high-level design
Step 1 of 4
A website, an API, two stores and a crawler fleet
Start with what the product needs on day one. The website and the extension call one API service. It reads history from a price store keyed by product and time, with daily rollups for long ranges and a cache for popular charts, and it writes subscriptions to a subscriptions table. A crawler fleet fetches product pages from the retailer and writes what it sees into the price store. This is enough for a demo and it names the two stores whose shape the rest of the design depends on. Its weakness is the crawler: nothing decides which product to fetch next, and a point is written on each check whether or not the price moved, so storage grows with checks rather than with changes.
What breaks first
The deep dives
Tracking 500 million products. The crawl budget covers a fraction of 500 million products at a time. Which products does the fleet check, how often, how does it pick the next one, and what changes when the retailer starts throttling you?
A concept that helps here: Budget pacing
A concept that helps here: Rate limiting algorithmsTrusting prices from the extension. An extension user posts a fake low price for a product with thousands of subscribers. Follow that price through the system and show why it reaches neither the chart nor anyone's inbox.
A concept that helps here: Reconciliation against an external system
A concept that helps here: Consistency modelsFanning out price drop alerts. A product with a huge subscriber list drops in price. How does each subscriber hear within the hour, what happens when a worker dies halfway through the list, and how do you avoid sending the same alert twice when the price wobbles back and forth across a threshold?
Concepts this guide uses
- Job scheduling and delayed work
- Budget pacing
- Rate limiting algorithms
- Reconciliation against an external system
- Consistency models
- Fan-out on write and on read
- Idempotency keys
- Delivery semantics: at most, at least, exactly once
Sources
- Wikipedia: Camelcamelcamel
- Wladimir Palant: Data exfiltration in Keepa Price Tracker
- Amazon Associates: Creators API documentation, API Rates and Limits
- The Register: Amazon algorithms price bio book at an astronomical sum
- Marc Brooker: Jitter: Making Things Better With Randomness
- Svix: How to Build a Scheduled Queue in Redis