Skip to content

How to design a price drop tracker

The first sections are free. The deep dives, level expectations and quiz are part of Pro.

On this page
Asked at
Practice this question

The problem

A price drop tracker watches the prices of hundreds of millions of retail products, draws each one's history as a chart, and emails people when a product falls under the number they chose. Two facts shape the answer. There are far more products than you can afford to check often, so a scheduler has to decide which products deserve a fresh look and which can wait. And the second source of prices, a browser extension running in a million strangers' browsers, is free and current but easy to lie through, so its reports need checking before they reach the chart or an inbox. Alerts have an hour to land, which leaves room for a queue and batches rather than a real-time path.

Clarifying questions

AskWhy it mattersA typical answer

How many products do we track, and how fresh does each price need to be?

The catalog size against the freshness target decides whether one crawl loop is enough or whether a scheduler has to ration checks across products.

About 500 million products. Prices may be briefly stale, and an alert must go out within 1 hour of the price changing, so products with subscribers need to be checked often enough for that.

Where do prices come from: an official API, our own crawlers, a browser extension, or all of them?

Each source has a different cost and a different trust level. A crawler fleet is under our control but has a budget the retailer can squeeze; extension reports are free but come from strangers.

Both a crawler fleet and the browser extension. The crawl budget covers a fraction of the catalog at a time, so the extension matters as a second source.

How many people run the extension, and do we trust what it sends?

A million clients that can post prices is a fan-in problem and an abuse problem at the same time. Knowing the count sizes the ingest path; knowing the trust answer decides whether reports need confirmation.

About 1 million active extension users. Treat what they send as unverified input: nothing from the extension should change the chart or fire an alert until something else agrees with it.

Does the chart need each recorded price, or is a daily view fine for long ranges?

It decides whether the read path serves raw points or precomputed daily summaries, which is the difference between a cheap query and a slow one on products with years of history.

History queries must answer in under 500 ms. A daily summary for long ranges and raw points for recent ranges is acceptable.

How bad is a late alert, a missed alert, or a duplicate?

The answer sets the delivery semantics. If a duplicate is merely annoying and a miss is a broken promise, you build for at-least-once delivery and add a record that suppresses repeats.

Alerts should arrive within the hour. A duplicate is annoying but tolerable; a missed alert is the failure people notice. Prices may be briefly stale: availability wins over consistency.

One retailer or many?

One retailer keeps the product identity simple (one id per product) and keeps the crawler fleet one kind of client. Several retailers means a product matching problem that is a design of its own.

Just Amazon. Comparing prices across retailers is out of scope.

The requirements menu

RequirementScopeWhy

View the price history of an Amazon product on the website or in the browser extension

Core

The read path and the reason people come. It drives the price store's shape and the under 500 ms target.

Subscribe to a price drop alert with a threshold

Core

The write path people care about most. The threshold is what the alert matcher later queries, so it belongs in the subscription row and in an index.

Get notified when the price falls below the threshold

Core

The product's promise. It forces a path from a detected change to an email or push within the hour.

Keep prices current for every tracked product

Core

The hard one. Current for 500 million products is a budget problem, and the scheduler that rations the budget is the center of the design.

Track separate prices per condition (new, used, third-party sellers)

Optional

Real trackers draw several lines per product. Take it if asked: it multiplies price points per product but changes nothing structural, since each line is a separate series keyed by product and condition.

Add a product to tracking the first time someone looks it up

Optional

Worth a sentence: an unknown product id from a search or an extension visit becomes a product row with a first check. It is how the catalog grows, and it costs one crawl per new product.

A daily digest instead of one email per drop

Optional

Cheap to add once alerts flow through a queue: hold matched alerts per user and send them together. Leave it unless asked, since the hour target is about single alerts.

Alerts on availability (back in stock)

Optional

Same pipeline as price alerts with a different predicate. Mention it as a variant and move on.

Predicting whether a price will fall further

Out of scope

A modeling problem on top of the history store. Name it as a consumer of the data and leave it.

Searching and discovering products

Out of scope

Search is its own system. In this round a product is found by its id or by pasting a retailer link.

Comparing prices across retailers

Out of scope

It needs product matching across catalogs, which is a different problem from tracking one catalog well.

Reviews and ratings

Out of scope

Nothing in the price pipeline depends on them. Skip.

Non-functional requirements

RequirementWhat it forces

Handle 500 million products

A product table and a schedule of that size, and a crawl budget that is a small fraction of the catalog per hour. The scheduler has to rank products rather than loop over them, and the price store has to grow with price changes rather than with checks.

Price history queries answer in under 500 ms

The chart reads one product's points in time order, so the price store is keyed by product and time, long ranges come from daily summaries computed ahead of time, and popular products' charts sit in a cache.

Alerts go out within 1 hour of a price change

Detection and delivery are decoupled by a queue so a long subscriber list does not stall the crawler, and the matcher finds matching subscriptions by index rather than by scan. The budget is generous enough that batching, not a push connection, is the right tool.

Availability over consistency: prices may be briefly stale

The read path can serve a cached chart and a slightly old current price. Writes to the price store and alert delivery can be asynchronous and retried, which means the alert path must tolerate at-least-once delivery.

Estimates

QuantityWorkingResultWhat it sizes

Checks per second if each product is checked on one shared interval

Once a day: 500,000,000 products ÷ 86,400 seconds ≈ 5,787 checks a second. Once an hour: 500,000,000 ÷ 3,600 ≈ 138,900 checks a second

About 5,787 a second for a daily sweep; about 138,900 a second for an hourly one

The crawler fleet under a fixed interval, and why an hourly sweep of the whole catalog is out of reach of any polite crawl budget.

Checks per second when the interval follows demand (an example split, not a target)

Say 1% of products are hot (have subscribers or traffic): 5,000,000 checked hourly = 5,000,000 ÷ 3,600 ≈ 1,389 a second. The other 495,000,000 checked weekly = 495,000,000 ÷ 604,800 ≈ 818 a second. Total ≈ 2,200 a second

About 2,200 checks a second, under half the daily sweep, while the hot 1% is checked 24 times as often as under a daily sweep

The fleet when checks follow demand, and the argument for a scheduler: freshness where it matters for less than a flat daily sweep.

Extension reports per second

1,000,000 users × 20 product pages a day (an assumption) = 20,000,000 reports a day ÷ 86,400 ≈ 231 a second

About 231 reports a second on average

The report ingest path and the per-user rate limit: a free source worth about a tenth of the demand-driven crawl rate, concentrated on popular products.

Confirmation crawls triggered by reports

If 1 in 100 reports looks like a big drop: 231 ÷ 100 ≈ 2.3 confirmation crawls a second. A flood of 10,000 fake reports for one product in a minute collapses to 1 crawl when confirmations are deduplicated per product

A few crawls a second in normal use; 1 per product under a flood

The cost of confirming reports with a crawl, and why confirmations must be coalesced per product.

Price store growth: write on change against write on check

Assume a point of product id, price and time takes about 20 bytes of raw payload, before indexes and database overhead (an assumption). Write on change, assuming 50 changes per product a year: 500,000,000 × 50 × 20 bytes ≈ 500 GB a year. Write each daily check: 500,000,000 × 365 × 20 bytes ≈ 3.65 TB a year

About 500 GB a year against about 3.65 TB a year

Why the detector writes a point when the price differs from the last one: storage grows with how often prices move, and the chart loses nothing since a flat line needs no extra points.

Send rate for a product with a huge subscriber list

Assume 100,000 subscribers on one product (a tenth of the extension's users, an assumption). 100,000 alerts within 3,600 seconds ≈ 28 sends a second. In batches of 100 that is 1,000 batches

About 28 sends a second, or 1,000 batches of 100, spread over the hour

The notification workers and the send queue: the hour budget is generous, so batching is enough and a real-time path is not needed.

Core entities

EntityWhat it holds

Product

One row per tracked retailer product: its current price, when it was last checked, and how urgently it should be checked again.

Price point

A confirmed price for a product at a moment, written when the price differs from the previous one.

Daily rollup

One row per product per day with low, high and close, so long chart ranges read a few hundred rows instead of each point.

Subscription

A user's request to hear when a product falls to or under a threshold.

Price report

An unconfirmed price seen by the extension during a real visit, held until a crawl or enough other reporters agree.

Sent alert

A record that one subscription was alerted for one price point, so a retried batch does not send the same alert twice; the subscription's armed flag handles bouncing prices.

User

Whoever subscribes or runs the extension. Accounts are ordinary; the field that matters here is the trust score.

The API

EndpointNotes

GET /products/{id}/price-history?period= → price points

Open to anyone in this design, since the chart is the product's front door. Short periods read raw points by product and time; long periods read the daily rollup rows instead, which is how the under 500 ms target holds for a product with years of history. Popular products' responses sit in a cache keyed by product and period and are refreshed when the detector confirms a new point. Pending extension reports are not in this response.

POST /subscriptions { product_id, threshold } → { subscription_id }

In this design the caller must be signed in, and the user id comes from the session, not the body. Writes the subscription row, bumps the product's subscriber count (which raises its crawl priority), and if the product is unknown creates it and schedules a first check. The threshold is stored as integer cents. If the current confirmed price is already at or under the threshold, say so in the response rather than sending an alert.

DELETE /subscriptions/{id}

Restricted to the owner, checked against the session. Deletes the row and lowers the product's subscriber count. A batch already on the send queue may still reach this user once; the worker re-reads the subscription before sending and skips a deleted one, which closes most of that window.

POST /price-reports { product_id, price } (from the extension) → 202

The reporter id comes from the extension's session, which is what the per-user rate limit and the trust score key on. 202 is deliberate: the price is accepted for checking, not recorded as truth. The service compares it with the last confirmed price and either holds it as pending, schedules a confirming crawl for a big drop, or drops it when the reporter's score is low or the limit is hit. The response does not say which of those happened, which keeps a bad actor from reading the checks straight off the reply.

The high-level design

subscribepriceWebsiteAPISubscriptionsCrawlersPrice store

Step 1 of 4

A website, an API, two stores and a crawler fleet

Start with what the product needs on day one. The website and the extension call one API service. It reads history from a price store keyed by product and time, with daily rollups for long ranges and a cache for popular charts, and it writes subscriptions to a subscriptions table. A crawler fleet fetches product pages from the retailer and writes what it sees into the price store. This is enough for a demo and it names the two stores whose shape the rest of the design depends on. Its weakness is the crawler: nothing decides which product to fetch next, and a point is written on each check whether or not the price moved, so storage grows with checks rather than with changes.

historysubscribeduefetchpricechangedeventconfirmedWebsiteAPIPrice storeSubscriptionsSchedulerCrawlersAmazonDetectorEvent queueAlert workersEmail and pushExtensionChecker
The whole design

What breaks first

The deep dives

Tracking 500 million products. The crawl budget covers a fraction of 500 million products at a time. Which products does the fleet check, how often, how does it pick the next one, and what changes when the retailer starts throttling you?

A concept that helps here: Job scheduling and delayed workA concept that helps here: Rate limiting algorithms

Trusting prices from the extension. An extension user posts a fake low price for a product with thousands of subscribers. Follow that price through the system and show why it reaches neither the chart nor anyone's inbox.

A concept that helps here: Rate limiting algorithmsA concept that helps here: Consistency models

Fanning out price drop alerts. A product with a huge subscriber list drops in price. How does each subscriber hear within the hour, what happens when a worker dies halfway through the list, and how do you avoid sending the same alert twice when the price wobbles back and forth across a threshold?

A concept that helps here: Fan-out on write and on readA concept that helps here: Idempotency keysA concept that helps here: Delivery semantics: at most, at least, exactly once

Concepts this guide uses

Sources

  1. Wikipedia: Camelcamelcamel
  2. Wladimir Palant: Data exfiltration in Keepa Price Tracker
  3. Amazon Associates: Creators API documentation, API Rates and Limits
  4. The Register: Amazon algorithms price bio book at an astronomical sum
  5. Marc Brooker: Jitter: Making Things Better With Randomness
  6. Svix: How to Build a Scheduled Queue in Redis
Sign in to report a mistake