Skip to content

Time-series storage

How stores built for timestamped measurements work: append-mostly writes, partitions by time, delta compression, rollups to coarser resolutions and expiry by age. In a design round size one by series count (memory), sample rate and retention (disk).

On this page

What it is

A time series is one measured quantity identified by a name and a small set of labels (host, region, sensor), with a stream of timestamp and value pairs behind it. The data has a distinctive shape: points arrive roughly in time order, are rarely changed once written, and are read as ranges ("this series over the last six hours") rather than as single rows. Reads cluster on the recent past: Facebook built Gorilla to hold the latest 26 hours of its monitoring data in memory, as a write-through cache in front of the HBase store that kept the history, and its authors name prioritizing recent data over historical data as their first lesson. Time-series storage is the set of techniques that fit this shape: partition by time, compress neighboring points against each other, roll raw points up into coarser summaries, and expire by age.

The write path, in one metrics database. The Prometheus storage documentation describes its local store like this. Incoming samples land in an in-memory block, and each one is also appended to a write-ahead log so a crash doesn't lose them. Each two-hour window of samples is sealed to disk as a directory holding compressed chunks, an index that maps metric names and labels to series, and a metadata file. In the background, those small blocks are merged into larger ones that cover at most a tenth of the retention window, capped at 31 days. With no time or size limit configured, samples are kept for 15 days. Two things to notice: new samples go to the log and the in-memory block, and expiring old data means removing whole blocks.

The same idea inside a relational database. TimescaleDB turns a Postgres table into a hypertable: rows are routed into chunks by timestamp, and the chunk interval is 7 days unless you change it. The vendor's tuning guide explains that the other features hang off the chunk: compression runs chunk by chunk, retention policies drop whole chunks, and continuous aggregates (precomputed rollups) default to a chunk interval 10 times the parent table's. Its sizing advice is to keep the chunks currently receiving writes, across all hypertables, within roughly a quarter of the database's memory. Larger chunks tend to compress better; smaller ones let you drop raw data sooner once a rollup has been materialized.

Why it compresses so well. Neighboring points in one series look alike, and the Gorilla paper describes two encodings that use that. Timestamps are stored as the delta of deltas: when samples arrive at a fixed interval, the change between consecutive gaps is usually zero, and about 96% of timestamps fit in a single bit. Values are XORed with the previous value; an unchanged value gives a zero XOR and is stored as one bit, and otherwise just the differing bits are kept, so roughly 51% of values also end up as a single bit. The result was an average 12x reduction. At launch, 26 hours of data fit in 1.3 TB of RAM spread over 20 machines. Eighteen months later, in spring 2015, the system was taking about 12 million points a second across more than 2 billion series, and the clusters had been doubled twice. The open-source successor, Beringei, reports compressing real-world monitoring data by more than 90 percent with the same scheme.

Rollups and expiry. Keeping all raw points indefinitely is rarely worth paying for, so stores pair retention rules with a downsampling job. Thanos documents a two-pass scheme in its compactor: raw samples older than 40 hours are rewritten at one point per 5 minutes, and 5-minute data older than 10 days at one point per hour. A downsampled chunk keeps a count, sum, min, max and counter aggregate for each interval, so a query over months reads far fewer samples and still computes correctly. Retention is set per resolution, and the documentation warns that if a resolution's retention is shorter than the age at which the next pass runs, data is deleted before it is downsampled. Two more points for a design: this kind of downsampling is for query speed rather than space (it adds blocks of similar size to the raw ones, roughly tripling storage), and once the raw resolution expires you lose the ability to zoom into history below the finest resolution you kept.

Series count is a capacity number that is easy to forget. Each distinct combination of label values is its own series, with its own index entry, memory and disk. The Prometheus instrumentation guide advises keeping a metric's label combinations below 10, and treats a metric past 100, or able to grow that far, as a signal to cut dimensions or move the analysis to a general-purpose processing system. Points per second, retention and bytes per sample size the disk; the series count sizes RAM and the index, and a label such as user ID or request ID grows without bound.

When to use it

Reach for a time-series store when the workload has the shape: a high, steady rate of timestamped points, reads that ask for ranges over one or a few series, and a clear rule for how long each resolution should live. Server metrics, sensor readings, price ticks and per-minute usage counters fit. The payoff is largest when you can set three knobs up front: a partition interval sized so the partitions still receiving writes sit in memory, a retention window per resolution, and the rollups you will need later, which must be in place before the raw data expires.

Describe the shape before you name a product. In a round, say what one point looks like (labels, timestamp, value), estimate points per second and the number of series, and then pick the store. For a metrics store you can size disk in your head: retention in seconds times samples per second times bytes per sample, and the documentation puts the last figure at 1 to 2 bytes on average. If the interviewer pushes on reads, say which window is hot and whether it should live in memory, as the Gorilla team did with its most recent 26 hours.

When not to

A low write rate often doesn't need most of this. If a service emits a handful of points a minute, an ordinary table with an index on series and timestamp, and a periodic delete of old rows, is often simpler to run and easier to query with joins. The partitioning machinery pays off most once deletes and index maintenance on a big table start to hurt, though a retention rule and time-bucketed queries can be worth having at any size.

Records that get updated, joined and fetched by identity (orders, tickets, profiles) belong in a transactional database even though they carry timestamps. A time-series store is tuned for appending points and scanning ranges, not for frequent changes to one record. Logs and traces are often served by their own stores.

Per-user or per-request analysis is the wrong job for a metrics store. Once a metric's label combinations are past about 100, the instrumentation guidance points you toward a general-purpose processing system; a warehouse or an event log keyed by user is the right home for that question, with the metrics store keeping the bounded dimensions.

Compared

OptionGood atPays for it withRight choice when

Plain indexed table

Simplicity, joins, updates

Row-by-row deletes and index growth as the table grows

The write rate is low, or points must be joined with transactional data

Partitioned relational (TimescaleDB hypertable)

SQL and joins plus chunk drops, per-chunk compression and continuous aggregates

Running Postgres at volume and tuning the chunk interval

You want SQL and joins with business tables and still need expiry by partition

Dedicated metrics store (sealed blocks, local retention)

Cheap appends, compact samples, built-in expiry and compaction

Bounded label cardinality and limited local retention

Infrastructure metrics with a small, known set of label values

In-memory hot tier over a disk store

Fast reads of the last day or so

RAM cost and a second system to operate

Dashboards and alerts dominate and most queries hit the recent past

Columnar warehouse or object store

Ad hoc analysis across many dimensions and cheap long-term keeping

Suited to analysis over long ranges more than second-by-second alerting

Long-term and high-cardinality questions, not live monitoring

In a picture

sealcompactdownsampleage outWriterIn-memory headWrite-ahead log2-hour blockLarger block5-min rollupsDropped
Points are appended to memory and a log, sealed into immutable blocks, merged, rolled up into coarser series, and dropped by age.

How to say it in an interview

The common mistake

Quiz

A store groups points into time blocks. What makes expiring old data cheap?
Which label is most likely to blow up the number of series in a metrics store?
Raw 10-second points are kept for two days; after that, 5-minute averages remain for a year. Which question about last month can you still answer?
Why did Facebook put an in-memory tier in front of its disk-based monitoring store?

Questions that use this

Related

Sources

  1. Prometheus documentation (3.16): Storage
  2. Prometheus documentation: Instrumentation best practices
  3. Tiger Data (Timescale) blog: Timescale Tips: Testing Your Chunk Size
  4. Thanos documentation: Compactor (compaction, retention and downsampling)
  5. The Morning Paper: Gorilla: A fast, scalable, in-memory time series database
  6. Engineering at Meta: Beringei: A high-performance time series storage engine
Sign in to report a mistake