Hot keys and hot partitions
One key that draws far more traffic than the rest, and the partition that holds it, saturated while its neighbors idle. More nodes don't spread one key; the fix is in the key or the path to it.
What it is
A partitioned store, cache, or log spreads keys across partitions, usually by hashing the key or by splitting the key space into ranges. Each partition has a ceiling: how many reads and writes per second one node or one shard can serve. A hot key is a key whose traffic is a large multiple of the typical key's. A hot partition is the one that holds that key, or a hot range of keys, and so receives far more traffic than the others. The store as a whole may have plenty of headroom while the one partition throttles or slows.
The reason more nodes don't help is the routing itself. Kafka's default partitioner, as the Confluent Platform producer documentation describes it, hashes a non-empty key with murmur2 and takes the remainder by the partition count, so messages with one key land on one partition. DynamoDB throttles a partition that receives more than 3,000 read or 1,000 write operations per second, and adaptive capacity can raise a hot partition's share up to that partition maximum and no further. Doubling the cluster leaves the hot key on a single partition with the same ceiling.
| Shape of the heat | Typical cause | What saturates |
|---|---|---|
Read-hot key | A viral item, a celebrity profile, a popular config row | The node or shard holding that key; the network link to it |
Write-hot key | A global counter, a hot feed, a per-tenant total | The single partition's write limit, however much capacity the table as a whole has |
Hot key range | A timestamp or auto-increment as the leading key part | Whichever server holds the newest range; the hotspot moves with the end of the key space |
The range case is the sneakiest because it moves. In Spanner, a primary key that increases monotonically, such as a timestamp, sends each new row to the end of the key space, so a single server takes the whole insert stream. When that split grows and a new split is created, the next server inherits the heat. The hot split statistics view labels this a moving hot spot and notes that load-based splitting doesn't resolve it; the schema advice is a UUID or a hash prefix in front of the key.
When to use it
Reach for these techniques when you can name the skew. A read-hot key with data that changes rarely is the easiest case: keep a copy in each application server's memory. Redis calls this client-side caching and builds it on the observation that a small share of the data is accessed often; since version 6 the server tracks which keys each connection has read and sends an invalidation when one is written, and the guidance is still to put a maximum lifetime on each local entry. The result is that most requests for the hot key are answered inside the app server.
A stampede of identical misses is the companion problem: when the cached copy expires, hundreds of concurrent requests all miss and all go to the store at once. Request coalescing fixes this. Go's singleflight package, for example, lets one call for a key run while duplicate callers wait and receive its result, so the store sees one load instead of hundreds. A good local cache uses coalescing on its miss path.
A write-hot key can't be solved by caching, because each write must land. Write sharding appends a suffix to the partition key, either random (date.3) or calculated from an item attribute (item_id % N), so the writes arrive as N separate keys and are usually spread across several partitions rather than one. The cost moves to reads: with a random suffix a reader queries all N shards and merges; with a calculated suffix a reader who knows the attribute can go straight to one shard.
A hot or unbounded range calls for changing the key itself. Cassandra 4.0's data modeling guide describes bucketing: add a coarse time unit such as a month to the partition key so each partition stays a moderate size and recent writes rotate to a new bucket. For sequential keys at a high write rate, put a hash or a UUID in the leading position so inserts scatter across servers.
When not to
Don't shard a key that fits. If a single DynamoDB item sees 400 writes per second, it's under the 1,000 per second that adaptive capacity can give one item on its own partition, and write sharding would turn each read of the total into N queries for nothing. The same applies to a short spike: burst capacity holds up to 300 seconds of unused read and write capacity for exactly that, and a brief viral moment may pass before any redesign would ship.
Don't split a key whose order matters. In Kafka, the per-key routing is what lets a consumer see one key's messages in the order they were produced (given producer settings that keep retries from reordering them); spread a hot key across partitions and that ordering is gone, so a hot Kafka key is often better handled by a faster consumer or by batching than by re-keying. Likewise, hashing a timestamp key removes the cheap range scan over recent rows; if the table is read by time range and written slowly, keep the natural key.
Don't put a local cache in front of a read that has to be exact: a seat count, a balance, a rate limit decision. A per-server copy can lag until its lifetime ends or an invalidation arrives, and with N app servers there are N copies lagging independently. Send those reads to the store and spend the effort on keeping them cheap instead.
Compared
| Remedy | Fixes | Costs | When the simpler option is right |
|---|---|---|---|
In-process cache per app server | Read-hot key | Stale copies until lifetime or invalidation; memory on each server | The read must be exact, or the hot key changes faster than it is read |
Request coalescing | Stampede of identical misses | Callers wait on one load; one failure fails them all | Misses are spread over many distinct keys, so there's little to merge |
Write sharding with a suffix | Write-hot key or counter | Reads fan out to N shards and merge | Writes stay under one partition's limit (1,000 per second on the platform above) |
Bucketing by time | Partition that grows without bound | Queries that span buckets issue several reads | Per-key volume is small enough that one partition stays moderate |
Hash or UUID as leading key | Moving hot spot from sequential keys | Loses the range scan by time | Insert rate is low and the table is read by time range |
Rely on the platform (burst, adaptive capacity) | Mild or brief skew | Capped at the per-partition maximum | The hot item stays under 3,000 reads and 1,000 writes per second |
In a picture
Step 1 of 4
One key, one partition
Requests for the hot key all hash to partition A. Partition B sits idle, and adding a partition C wouldn't change where the key lands.
How to say it in an interview
The common mistake
Quiz
Questions that use this
Related
Sources
- Amazon DynamoDB Developer Guide: DynamoDB burst and adaptive capacity
- aws-samples/aws-dynamodb-examples: Write Sharding in DynamoDB
- Confluent Platform documentation: Kafka Producer
- Redis documentation: Client-side caching reference
- Go package documentation: golang.org/x/sync/singleflight
- Spanner documentation: Schema design best practices
- Spanner documentation: Split statistics
- Apache Cassandra 4.0 documentation: Evaluating and Refining Data Models