How to design a metrics monitoring system
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Collecting the metrics
- Storing a year of time series
- Alerts that page once
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
A metrics monitoring system takes a steady stream of numbers from each machine (CPU load, memory, requests a second), keeps them as time series, charts them, and pages someone when a rule trips. The load is lopsided: a hundred thousand machines write without pause, while reads are rare and mostly about the last day. The answer turns on three choices: a write path that keeps collecting while the store is down (collectors feeding a log, then a store built for time data), a storage plan that keeps raw points for a week and coarser rollups for a year, and an alert path that pages once per incident and keeps retrying until the page lands.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
How many machines report, and how many metrics does each one expose? | The series count sets the write rate, the storage size and the number of store nodes. It is the number the rest of the round hangs on. | About 100,000 machines in 1,000 pools of 100, each exposing about 100 metrics, so about 10 million time series. Collection interval is your call: say it out loud, because the write rate follows from it. |
Do we control the servers we monitor, and do any of them live for less than a collection interval? | It decides whether collectors can go and read each server (pull) or whether servers have to send their numbers in (push), and whether both paths are needed. | They are our own servers, and most are long-lived. A few batch jobs finish in seconds and some agents sit behind networks the collectors can't reach, so expect to defend one model and know what the other buys. |
How long do we keep data, and at what resolution? | Retention times storage per point is the storage bill, and the resolution tiers decide whether a downsampling job exists at all. | A year in total: raw points for 7 days, then 1-minute resolution to 30 days, then 1-hour resolution to a year. |
Who reads the data, and how fresh and how fast do the charts need to be? | If most reads land on recent data, the last day belongs on fast storage or in memory while older tiers can be slower and cheaper. | Engineers on dashboards and during incidents, plus the alert evaluator. About 85% of queries ask for the last 26 hours, and those should come back fast; a chart over a full year may take longer. |
Which alert channels, and how strict is delivery? | At-least-once delivery means retries, and retries mean duplicates unless something dedups. The answer decides whether you need alert state at all. | Email, phone, PagerDuty and webhooks. A critical alert must reach its channel at least once; a rare duplicate is tolerable, a missed page is not. |
Is dropping an occasional data point acceptable? | If yes, collectors can send without waiting for an acknowledgment and the pipeline can shed load; if no, you need acknowledgments end to end, which is a much heavier design. | Yes for metrics: a missing point in a chart is a gap, not an outage. Alerts are the exception and get their own delivery path. |
Do we build dashboards and the paging front end, or buy them? | Grafana and PagerDuty exist. Spending the round on a chart renderer leaves no time for the pipeline and the store, which is where the hard problems are. | Buy them. Spend the time on collection, storage and the alert path, and say why. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
About 10 million metrics: 1,000 pools of 100 machines, about 100 metrics each | At a 15-second interval that is about 667,000 points a second arriving without pause. The store has to be sharded from the start, a log in front of it absorbs a slow or dead node, and labels have to stay bounded so the series total stays near 10 million rather than drifting past it. |
One-year retention: raw for 7 days, 1-minute resolution to 30 days, 1-hour after | A downsampling job that turns raw points into minute and hour rollups, and a cold tier for the oldest data. Sizing shows raw data at full resolution for a year would be far too large, so the tiers are not optional. |
Recent data is fast to read: about 85% of queries ask for the last 26 hours | The last day lives on fast storage or in memory, behind a query cache shared by dashboards and alert evaluation. Older tiers can sit on slower disks or object storage without most users noticing. |
No missed critical alert: each one reaches its channel at least once | Alert events go through a durable log to workers that keep retrying a channel until it accepts, and alert state is kept so those retries don't page the same incident twice. The metrics path may drop a point; the alert path is built so that an event survives the failures we plan for: a provider timeout, a worker crash, a manager restart. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Write rate at a chosen collection interval | 10 million series ÷ 15 seconds ≈ 667,000 points a second; at 10 seconds, 1 million a second; at 60 seconds, about 167,000 a second | About 667,000 points a second at a 15-second interval, the figure the rest of the guide uses | Everything downstream: the number of store nodes, the log's buffer, the raw tier. Say the interval before you say the rate. |
Store nodes for the raw writes | 667,000 ÷ 250,000 writes per second per node ≈ 2.7, so 3 nodes at face value; × 2 for a replica of each shard = 6; run at about half capacity for headroom and incidents ≈ 12 | About 3 nodes on paper, about 12 in practice | The hot store cluster. Also why a single node is out: the InfluxDB sizing guide warns that a series count around 10 million can cause out-of-memory failures on one node, and that is the whole fleet here. |
The raw 7-day tier | 667,000 points a second × 604,800 seconds in a week ≈ 4.0 × 10^11 points. At 16 bytes a point (an 8-byte timestamp and an 8-byte value) ≈ 6.5 TB. At the twelvefold reduction Gorilla reports, about 1.3 bytes a point ≈ 540 GB; at the 1 to 2 bytes a sample Prometheus reports for its format, 400 to 800 GB | About 6.5 TB uncompressed, under 1 TB with time-series compression | The fast disks for the hot tier, and the case for delta-coded timestamps: compression is the difference between a few nodes' worth of SSD and a rack of it. |
The rollup tiers for a year | 1-minute for 30 days: 10 million × 43,200 minutes ≈ 4.3 × 10^11 points. 1-hour for a year: 10 million × 8,760 hours ≈ 8.8 × 10^10 points. Together about 5.2 × 10^11 points: ≈ 8.3 TB at 16 bytes, about 1 TB compressed, before multiplying by the count, sum, min and max a rollup usually keeps | About the same point count as one raw week, for a whole year | Why downsampling works: the minute tier for a month is about as many points as the raw tier for a week, and the hour tier for a year is a fifth of that. |
Log buffer for a ten-minute store outage | 667,000 points a second × 600 seconds ≈ 400 million points; at about 100 bytes a point on the wire (name, labels, timestamp, value) ≈ 40 GB | About 40 GB of Kafka retention per ten minutes the store is down | How much the log must hold before the store catches up, and how long a retention to set so a longer outage still loses nothing. |
Scrapes a second across the fleet | 100,000 machines ÷ 15 seconds ≈ 6,667 scrapes a second. Spread over 100 collectors, each owns about 1,000 targets and does about 67 scrapes a second, producing about 6,667 points a second | About 6,667 scrapes a second in total, about 67 per collector | The collector tier: a modest per-collector load, so the hard part is assigning targets, not raw throughput. |
Alert rule evaluations | 125,000 rules ÷ 60 seconds ≈ 2,083 rule queries a second against the query service, if each rule runs once a minute | About 2,083 queries a second from alerting | The read load alerting adds to the query service, and why alert evaluation is sharded across workers rather than run from one process. |
Series created by one user-id label | 100 million daily users × 1 metric carrying a user_id label = 100 million series, ten times the 10 million the whole fleet produces | One label on one metric can outgrow the entire system | Why label values stay bounded (host, pool, region, status code) and why ingestion rejects or drops labels with unbounded values. |
Core entities
| Entity | What it holds |
|---|---|
Time series | One metric name plus one exact set of label values, with the timestamped points that belong to it. |
Alert rule | What to check, how long it must stay true, and who hears about it. |
Alert | One rule tripping on one series, with its lifecycle from pending to resolved. |
Dashboard | A saved page of panels, each one a query and a chart type. |
The API
| Endpoint | Notes |
|---|---|
| Served by the application or a host agent. A collector that gets a connection error here has learned something useful: the target is down. That |
| Behind a load balancer, for jobs that finish before a collector would reach them and for agents the collectors can't reach. Answer |
| The one read path, used by dashboards and by alert evaluation. The service picks the tier by range: raw for the last week, minute rollups for the month, hour rollups beyond. Identical queries within a short window can be answered from a cache, which matters when dashboards re-run the same panel on a timer. |
| Idempotent by id, so a config deploy can re-send rules safely. Rules are validated here (does the expression parse, is the channel known) and loaded into the alert manager's rule cache on change. |
The high-level design
Step 1 of 4
Scrape, store, chart
Start with the smallest thing that charts a number. Each server exposes GET /metrics. Collectors read a list of targets and scrape each one on a fixed interval, then write the points into a store built for time data: points keyed by series and time, compressed against their neighbors. A query service answers GET /v1/query for the dashboards. At about 667,000 points a second from 10 million series, a single store node is already out: the InfluxDB sizing guide rates one node with 8 cores and 32 GB at above 250,000 writes per second, and warns that a series count around or above 10 million can cause out-of-memory failures even with plenty of RAM, so the store is a cluster sharded by series from the first drawing.
What breaks first
The deep dives
Collecting the metrics. A hundred thousand machines each expose about a hundred metrics. How does one data point travel from a server into the pipeline, how do the collectors divide the machines so each is read by exactly one of them, and what happens to points while the store is down?
Storing a year of time series. Ten million series write without pause and must be queryable for a year. What do you keep at full detail, what do you roll up and when, where does each tier live, and what keeps the series count from exploding?
A concept that helps here: Downsampling and rollups
A concept that helps here: Hot keys and hot partitionsAlerts that page once. A host stops reporting at 3 a.m. Walk the path from a stored rule to a page on someone's phone, and explain how the person is paged once even when two alert managers are running and the paging provider is slow.
Concepts this guide uses
Sources
- Alex Xu, System Design Interview volume two, metrics monitoring chapter (notes by Preslav Mihaylov)
- Prometheus documentation: Frequently asked questions (why pull rather than push)
- Prometheus documentation: When to use the Pushgateway
- Prometheus documentation: Storage
- Prometheus documentation: Alertmanager
- Prometheus documentation: Metric and label naming
- The Morning Paper (Adrian Colyer): Gorilla, a fast, scalable, in-memory time series database
- Uber Engineering: Observability at Scale, Building Uber's Alerting Ecosystem
- InfluxDB 1.1 documentation (archive): Hardware sizing guidelines
- Thanos documentation: Compactor (downsampling and retention)