Skip to content

Hot keys and hot partitions

One key that draws far more traffic than the rest, and the partition that holds it, saturated while its neighbors idle. More nodes don't spread one key; the fix is in the key or the path to it.

On this page

What it is

A partitioned store, cache, or log spreads keys across partitions, usually by hashing the key or by splitting the key space into ranges. Each partition has a ceiling: how many reads and writes per second one node or one shard can serve. A hot key is a key whose traffic is a large multiple of the typical key's. A hot partition is the one that holds that key, or a hot range of keys, and so receives far more traffic than the others. The store as a whole may have plenty of headroom while the one partition throttles or slows.

The reason more nodes don't help is the routing itself. Kafka's default partitioner, as the Confluent Platform producer documentation describes it, hashes a non-empty key with murmur2 and takes the remainder by the partition count, so messages with one key land on one partition. DynamoDB throttles a partition that receives more than 3,000 read or 1,000 write operations per second, and adaptive capacity can raise a hot partition's share up to that partition maximum and no further. Doubling the cluster leaves the hot key on a single partition with the same ceiling.

Shape of the heatTypical causeWhat saturates

Read-hot key

A viral item, a celebrity profile, a popular config row

The node or shard holding that key; the network link to it

Write-hot key

A global counter, a hot feed, a per-tenant total

The single partition's write limit, however much capacity the table as a whole has

Hot key range

A timestamp or auto-increment as the leading key part

Whichever server holds the newest range; the hotspot moves with the end of the key space

The range case is the sneakiest because it moves. In Spanner, a primary key that increases monotonically, such as a timestamp, sends each new row to the end of the key space, so a single server takes the whole insert stream. When that split grows and a new split is created, the next server inherits the heat. The hot split statistics view labels this a moving hot spot and notes that load-based splitting doesn't resolve it; the schema advice is a UUID or a hash prefix in front of the key.

When to use it

Reach for these techniques when you can name the skew. A read-hot key with data that changes rarely is the easiest case: keep a copy in each application server's memory. Redis calls this client-side caching and builds it on the observation that a small share of the data is accessed often; since version 6 the server tracks which keys each connection has read and sends an invalidation when one is written, and the guidance is still to put a maximum lifetime on each local entry. The result is that most requests for the hot key are answered inside the app server.

A stampede of identical misses is the companion problem: when the cached copy expires, hundreds of concurrent requests all miss and all go to the store at once. Request coalescing fixes this. Go's singleflight package, for example, lets one call for a key run while duplicate callers wait and receive its result, so the store sees one load instead of hundreds. A good local cache uses coalescing on its miss path.

A write-hot key can't be solved by caching, because each write must land. Write sharding appends a suffix to the partition key, either random (date.3) or calculated from an item attribute (item_id % N), so the writes arrive as N separate keys and are usually spread across several partitions rather than one. The cost moves to reads: with a random suffix a reader queries all N shards and merges; with a calculated suffix a reader who knows the attribute can go straight to one shard.

A hot or unbounded range calls for changing the key itself. Cassandra 4.0's data modeling guide describes bucketing: add a coarse time unit such as a month to the partition key so each partition stays a moderate size and recent writes rotate to a new bucket. For sequential keys at a high write rate, put a hash or a UUID in the leading position so inserts scatter across servers.

When not to

Don't shard a key that fits. If a single DynamoDB item sees 400 writes per second, it's under the 1,000 per second that adaptive capacity can give one item on its own partition, and write sharding would turn each read of the total into N queries for nothing. The same applies to a short spike: burst capacity holds up to 300 seconds of unused read and write capacity for exactly that, and a brief viral moment may pass before any redesign would ship.

Don't split a key whose order matters. In Kafka, the per-key routing is what lets a consumer see one key's messages in the order they were produced (given producer settings that keep retries from reordering them); spread a hot key across partitions and that ordering is gone, so a hot Kafka key is often better handled by a faster consumer or by batching than by re-keying. Likewise, hashing a timestamp key removes the cheap range scan over recent rows; if the table is read by time range and written slowly, keep the natural key.

Don't put a local cache in front of a read that has to be exact: a seat count, a balance, a rate limit decision. A per-server copy can lag until its lifetime ends or an invalidation arrives, and with N app servers there are N copies lagging independently. Send those reads to the store and spend the effort on keeping them cheap instead.

Compared

RemedyFixesCostsWhen the simpler option is right

In-process cache per app server

Read-hot key

Stale copies until lifetime or invalidation; memory on each server

The read must be exact, or the hot key changes faster than it is read

Request coalescing

Stampede of identical misses

Callers wait on one load; one failure fails them all

Misses are spread over many distinct keys, so there's little to merge

Write sharding with a suffix

Write-hot key or counter

Reads fan out to N shards and merge

Writes stay under one partition's limit (1,000 per second on the platform above)

Bucketing by time

Partition that grows without bound

Queries that span buckets issue several reads

Per-key volume is small enough that one partition stays moderate

Hash or UUID as leading key

Moving hot spot from sequential keys

Loses the range scan by time

Insert rate is low and the table is read by time range

Rely on the platform (burst, adaptive capacity)

Mild or brief skew

Capped at the per-partition maximum

The hot item stays under 3,000 reads and 1,000 writes per second

In a picture

Reads of one keyHash of keyPartition APartition B

Step 1 of 4

One key, one partition

Requests for the hot key all hash to partition A. Partition B sits idle, and adding a partition C wouldn't change where the key lands.

How to say it in an interview

The common mistake

Quiz

One item in a hash-partitioned cache cluster is read 400,000 times a second; the other keys are quiet. Which change helps most?
A DynamoDB counter item receives 20,000 increments a second and throttles. What fixes it?
A range-partitioned table takes heavy inserts with a commit timestamp as the first key part. Why does load-based splitting not resolve the hotspot?
What does request coalescing change when a hot cache entry expires?

Questions that use this

Related

Sources

  1. Amazon DynamoDB Developer Guide: DynamoDB burst and adaptive capacity
  2. aws-samples/aws-dynamodb-examples: Write Sharding in DynamoDB
  3. Confluent Platform documentation: Kafka Producer
  4. Redis documentation: Client-side caching reference
  5. Go package documentation: golang.org/x/sync/singleflight
  6. Spanner documentation: Schema design best practices
  7. Spanner documentation: Split statistics
  8. Apache Cassandra 4.0 documentation: Evaluating and Refining Data Models
Sign in to report a mistake