Skip to content

How to design a metrics monitoring system

The first sections are free. The deep dives, level expectations and quiz are part of Pro.

On this page
Asked at
Practice this question

The problem

A metrics monitoring system takes a steady stream of numbers from each machine (CPU load, memory, requests a second), keeps them as time series, charts them, and pages someone when a rule trips. The load is lopsided: a hundred thousand machines write without pause, while reads are rare and mostly about the last day. The answer turns on three choices: a write path that keeps collecting while the store is down (collectors feeding a log, then a store built for time data), a storage plan that keeps raw points for a week and coarser rollups for a year, and an alert path that pages once per incident and keeps retrying until the page lands.

Clarifying questions

AskWhy it mattersA typical answer

How many machines report, and how many metrics does each one expose?

The series count sets the write rate, the storage size and the number of store nodes. It is the number the rest of the round hangs on.

About 100,000 machines in 1,000 pools of 100, each exposing about 100 metrics, so about 10 million time series. Collection interval is your call: say it out loud, because the write rate follows from it.

Do we control the servers we monitor, and do any of them live for less than a collection interval?

It decides whether collectors can go and read each server (pull) or whether servers have to send their numbers in (push), and whether both paths are needed.

They are our own servers, and most are long-lived. A few batch jobs finish in seconds and some agents sit behind networks the collectors can't reach, so expect to defend one model and know what the other buys.

How long do we keep data, and at what resolution?

Retention times storage per point is the storage bill, and the resolution tiers decide whether a downsampling job exists at all.

A year in total: raw points for 7 days, then 1-minute resolution to 30 days, then 1-hour resolution to a year.

Who reads the data, and how fresh and how fast do the charts need to be?

If most reads land on recent data, the last day belongs on fast storage or in memory while older tiers can be slower and cheaper.

Engineers on dashboards and during incidents, plus the alert evaluator. About 85% of queries ask for the last 26 hours, and those should come back fast; a chart over a full year may take longer.

Which alert channels, and how strict is delivery?

At-least-once delivery means retries, and retries mean duplicates unless something dedups. The answer decides whether you need alert state at all.

Email, phone, PagerDuty and webhooks. A critical alert must reach its channel at least once; a rare duplicate is tolerable, a missed page is not.

Is dropping an occasional data point acceptable?

If yes, collectors can send without waiting for an acknowledgment and the pipeline can shed load; if no, you need acknowledgments end to end, which is a much heavier design.

Yes for metrics: a missing point in a chart is a gap, not an outage. Alerts are the exception and get their own delivery path.

Do we build dashboards and the paging front end, or buy them?

Grafana and PagerDuty exist. Spending the round on a chart renderer leaves no time for the pipeline and the store, which is where the hard problems are.

Buy them. Spend the time on collection, storage and the alert path, and say why.

The requirements menu

RequirementScopeWhy

Collect operational metrics (CPU, memory, disk, requests per second) from every server

Core

The write path. Everything else in the design is downstream of how one data point leaves a machine.

Store them as time series and answer queries over any time range

Core

The store is the component that breaks first, and the query over a year decides whether rollups exist.

Fire alerts by email, phone, PagerDuty and webhooks when a rule matches

Core

The reason the system exists: a person is woken up. It brings its own delivery and dedup problems.

Show metrics and alerts on dashboards

Core

The read path. It shares a query service with alerting, which is why that service gets a cache.

Accept pushed metrics from jobs that finish before a collector gets to them, and from agents behind networks the collectors can't reach

Optional

Worth taking: it is a small endpoint behind a load balancer, and it shows you know where a pull model stops working. Leave it if the interviewer says each target is long-lived and reachable.

Downsample old data into coarser rollups and move the oldest to cheap storage

Optional

Take it: it is how a year of retention becomes affordable, and the sizing arithmetic makes the case for you.

Silence alerts during planned maintenance

Optional

Cheap to add once alert state exists, and worth naming because planned work otherwise pages people for outages they caused on purpose. Mention it; don't design it.

Group related alerts into a single notification

Optional

Take it if the alerting deep dive comes up: a rack losing power fires hundreds of host alerts, and one page is what the on-call wants.

Accept data points that arrive hours late and place them in the right time window

Optional

Comes up for devices on bad networks. It changes how the stream aggregation and the rollup jobs handle windows that were already closed, so raise it and park it unless asked.

Many isolated customer accounts, as a hosted monitoring product would need

Out of scope

Multi-tenancy adds per-tenant quotas and fairness across the pipeline. Name it as a later concern; the single-company version is already a full round.

Anomaly detection instead of fixed thresholds

Out of scope

A modeling problem on top of the same query service. Say where the model would plug in, then leave it.

Business metrics

Out of scope

Revenue and sign-ups come from the product database on a different cadence and with different accuracy needs. Keep this system for operational numbers.

Log search

Out of scope

Logs are text with high volume per event and need an inverted index, a different store from numeric time series.

Distributed tracing

Out of scope

Traces are per-request trees, sampled and joined by trace id. Related, but a separate pipeline and store.

Non-functional requirements

RequirementWhat it forces

About 10 million metrics: 1,000 pools of 100 machines, about 100 metrics each

At a 15-second interval that is about 667,000 points a second arriving without pause. The store has to be sharded from the start, a log in front of it absorbs a slow or dead node, and labels have to stay bounded so the series total stays near 10 million rather than drifting past it.

One-year retention: raw for 7 days, 1-minute resolution to 30 days, 1-hour after

A downsampling job that turns raw points into minute and hour rollups, and a cold tier for the oldest data. Sizing shows raw data at full resolution for a year would be far too large, so the tiers are not optional.

Recent data is fast to read: about 85% of queries ask for the last 26 hours

The last day lives on fast storage or in memory, behind a query cache shared by dashboards and alert evaluation. Older tiers can sit on slower disks or object storage without most users noticing.

No missed critical alert: each one reaches its channel at least once

Alert events go through a durable log to workers that keep retrying a channel until it accepts, and alert state is kept so those retries don't page the same incident twice. The metrics path may drop a point; the alert path is built so that an event survives the failures we plan for: a provider timeout, a worker crash, a manager restart.

Estimates

QuantityWorkingResultWhat it sizes

Write rate at a chosen collection interval

10 million series ÷ 15 seconds ≈ 667,000 points a second; at 10 seconds, 1 million a second; at 60 seconds, about 167,000 a second

About 667,000 points a second at a 15-second interval, the figure the rest of the guide uses

Everything downstream: the number of store nodes, the log's buffer, the raw tier. Say the interval before you say the rate.

Store nodes for the raw writes

667,000 ÷ 250,000 writes per second per node ≈ 2.7, so 3 nodes at face value; × 2 for a replica of each shard = 6; run at about half capacity for headroom and incidents ≈ 12

About 3 nodes on paper, about 12 in practice

The hot store cluster. Also why a single node is out: the InfluxDB sizing guide warns that a series count around 10 million can cause out-of-memory failures on one node, and that is the whole fleet here.

The raw 7-day tier

667,000 points a second × 604,800 seconds in a week ≈ 4.0 × 10^11 points. At 16 bytes a point (an 8-byte timestamp and an 8-byte value) ≈ 6.5 TB. At the twelvefold reduction Gorilla reports, about 1.3 bytes a point ≈ 540 GB; at the 1 to 2 bytes a sample Prometheus reports for its format, 400 to 800 GB

About 6.5 TB uncompressed, under 1 TB with time-series compression

The fast disks for the hot tier, and the case for delta-coded timestamps: compression is the difference between a few nodes' worth of SSD and a rack of it.

The rollup tiers for a year

1-minute for 30 days: 10 million × 43,200 minutes ≈ 4.3 × 10^11 points. 1-hour for a year: 10 million × 8,760 hours ≈ 8.8 × 10^10 points. Together about 5.2 × 10^11 points: ≈ 8.3 TB at 16 bytes, about 1 TB compressed, before multiplying by the count, sum, min and max a rollup usually keeps

About the same point count as one raw week, for a whole year

Why downsampling works: the minute tier for a month is about as many points as the raw tier for a week, and the hour tier for a year is a fifth of that.

Log buffer for a ten-minute store outage

667,000 points a second × 600 seconds ≈ 400 million points; at about 100 bytes a point on the wire (name, labels, timestamp, value) ≈ 40 GB

About 40 GB of Kafka retention per ten minutes the store is down

How much the log must hold before the store catches up, and how long a retention to set so a longer outage still loses nothing.

Scrapes a second across the fleet

100,000 machines ÷ 15 seconds ≈ 6,667 scrapes a second. Spread over 100 collectors, each owns about 1,000 targets and does about 67 scrapes a second, producing about 6,667 points a second

About 6,667 scrapes a second in total, about 67 per collector

The collector tier: a modest per-collector load, so the hard part is assigning targets, not raw throughput.

Alert rule evaluations

125,000 rules ÷ 60 seconds ≈ 2,083 rule queries a second against the query service, if each rule runs once a minute

About 2,083 queries a second from alerting

The read load alerting adds to the query service, and why alert evaluation is sharded across workers rather than run from one process.

Series created by one user-id label

100 million daily users × 1 metric carrying a user_id label = 100 million series, ten times the 10 million the whole fleet produces

One label on one metric can outgrow the entire system

Why label values stay bounded (host, pool, region, status code) and why ingestion rejects or drops labels with unbounded values.

Core entities

EntityWhat it holds

Time series

One metric name plus one exact set of label values, with the timestamped points that belong to it.

Alert rule

What to check, how long it must stay true, and who hears about it.

Alert

One rule tripping on one series, with its lifecycle from pending to resolved.

Dashboard

A saved page of panels, each one a query and a chart type.

The API

EndpointNotes

GET /metrics on each server → current values (pull model)

Served by the application or a host agent. A collector that gets a connection error here has learned something useful: the target is down. That up or down signal is the cheapest health check in the system.

POST /v1/metrics [{ name, labels, timestamp, value }] (push model)

Behind a load balancer, for jobs that finish before a collector would reach them and for agents the collectors can't reach. Answer 202 Accepted as soon as the batch is written to the log; the sender doesn't wait for the store. Reject label values that look unbounded here, before they become series.

GET /v1/query?expr=…&from=…&to=… → series

The one read path, used by dashboards and by alert evaluation. The service picks the tier by range: raw for the last week, minute rollups for the month, hour rollups beyond. Identical queries within a short window can be answered from a cache, which matters when dashboards re-run the same panel on a timer.

PUT /v1/alert-rules/{id} { expr, for, severity, channels }

Idempotent by id, so a config deploy can re-send rules safely. Rules are validated here (does the expression parse, is the channel known) and loaded into the alert manager's rule cache on change.

The high-level design

scrapeCollectorsServers: /metricsTime-series DBQuery serviceDashboards

Step 1 of 4

Scrape, store, chart

Start with the smallest thing that charts a number. Each server exposes GET /metrics. Collectors read a list of targets and scrape each one on a fixed interval, then write the points into a store built for time data: points keyed by series and time, compressed against their neighbors. A query service answers GET /v1/query for the dashboards. At about 667,000 points a second from 10 million series, a single store node is already out: the InfluxDB sizing guide rates one node with 8 cores and 32 GB at above 250,000 writes per second, and warns that a series count around or above 10 million can cause out-of-memory failures even with plenty of RAM, so the store is a cluster sharded by series from the first drawing.

targetsscrapepushkeyedolderstateeventsretryService registryCollectorsServers: /metricsPush agentsLoad balancerKafkaStream consumersHot TSDB: 7 daysDownsamplingjobsRollups: 1 yearQuery serviceQuery cacheDashboardsAlert managerAlert stateAlert eventsNotificationworkersPagerDuty,email
The whole design

What breaks first

The deep dives

Collecting the metrics. A hundred thousand machines each expose about a hundred metrics. How does one data point travel from a server into the pipeline, how do the collectors divide the machines so each is read by exactly one of them, and what happens to points while the store is down?

A concept that helps here: Consistent hashingA concept that helps here: Queues and logsA concept that helps here: Backpressure and load shedding

Storing a year of time series. Ten million series write without pause and must be queryable for a year. What do you keep at full detail, what do you roll up and when, where does each tier live, and what keeps the series count from exploding?

A concept that helps here: Time-series storageA concept that helps here: Hot keys and hot partitions

Alerts that page once. A host stops reporting at 3 a.m. Walk the path from a stored rule to a page on someone's phone, and explain how the person is paged once even when two alert managers are running and the paging provider is slow.

A concept that helps here: Delivery semantics: at most, at least, exactly onceA concept that helps here: Idempotency keysA concept that helps here: Retries and dead-letter queues

Concepts this guide uses

Sources

  1. Alex Xu, System Design Interview volume two, metrics monitoring chapter (notes by Preslav Mihaylov)
  2. Prometheus documentation: Frequently asked questions (why pull rather than push)
  3. Prometheus documentation: When to use the Pushgateway
  4. Prometheus documentation: Storage
  5. Prometheus documentation: Alertmanager
  6. Prometheus documentation: Metric and label naming
  7. The Morning Paper (Adrian Colyer): Gorilla, a fast, scalable, in-memory time series database
  8. Uber Engineering: Observability at Scale, Building Uber's Alerting Ecosystem
  9. InfluxDB 1.1 documentation (archive): Hardware sizing guidelines
  10. Thanos documentation: Compactor (downsampling and retention)
Sign in to report a mistake