Sharding and partition keys
Sharding splits one data set across several servers, and the partition key decides which shard holds each row. The key you pick shapes which queries can be routed to a few shards and which have to ask most of them.
What it is
Sharding is horizontal scaling for a data store: the data set and its load are divided across several servers, each holding a subset, and you add servers as load grows. The alternative, a bigger single machine, hits a practical ceiling set by the hardware on offer; sharding trades that ceiling for more infrastructure to run. The partition key (shard key, in some products) is the field or set of fields whose value decides which shard a row lands on. In DynamoDB the partition key part of the primary key selects the logical partition and through it the physical one. In Cassandra 4.0 a partition is the set of rows sharing one partition key value, a hash of that value picks where it lives, and all rows of a partition sit on the same set of replica nodes.
The common ways to turn a key into a shard are lookup, range, hash and geographic. A lookup (directory) strategy keeps a map from key to shard, often through virtual shards that can be remapped to new physical servers without touching application code; the cost is a map lookup on each request. Range sharding puts neighboring key values on the same shard, so a query for one month of orders reads one place, but when activity concentrates on adjacent keys one shard gets hot and rebalancing is hard. Hash sharding computes a hash of the key and spreads sequential values apart, which gives a better chance of even load with no map to maintain, though a single very frequent key value still lands on one shard; range scans on the key then touch many shards, and moving data when shards are added is hard without consistent hashing. Geographic sharding keys on a region attribute and hosts each shard in that region. Document stores that offer both a hashed and a ranged mode describe the same trade-off: hashed spreads a key that climbs over time, ranged lets the router target the shards that hold a range.
| Property of the key | What goes wrong | Example from the documentation |
|---|---|---|
Cardinality (distinct values) | Each distinct value lives in at most one chunk, so the count of distinct values caps the number of chunks and so the number of useful shards | A |
Frequency | If a few values hold most of the rows, their chunks become the bottleneck and can grow too big to split | A status code with a handful of possible values |
Monotonic growth | A value that rises with each insert routes each new write to the chunk at the top of the range, so one shard takes the whole write load | Creation date rounded to the day: a day's writes all land on one value |
Query coverage | A query carrying the key is routed to the shards that hold it; one without the key is broadcast to all shards, and that path doesn't scale linearly as you add shards | User ID where there are many users: a good spread and the usual lookup |
A key can be composite. In CQL, PRIMARY KEY ((a, b), c) makes a and b together the partition key and c a clustering column: rows with the same a and b share one partition, and inside it rows are kept in clustering order. That is the shape for "all messages of one chat, newest first": the chat is the partition, message time is the clustering column, and the coordinator needs to contact as few nodes as possible to read it. The flip side is that a busy partition is a hotspot for both reads and writes, because all of its rows share the same replica nodes, so partitions should be neither too large nor too small. Other stores express the same split as a partition key that picks the shard plus a row key that orders items inside it.
A strong answer also says what happens when the key turns out wrong. Changing it means moving data, and how painful that is depends on the strategy: with a lookup map you remap virtual shards to new servers and the application doesn't notice; with plain hashing you need consistent hashing or a reshuffle of most rows. Some stores do the move for you. MongoDB 5.0 added reshardCollection, which changes a collection's shard key and redistributes its data, and refineCollectionShardKey adds suffix fields to an existing key. From 7.0, analyzeShardKey reports metrics for a candidate key based on sampled queries, so you can judge it before you commit. Even so, a shard can take part in one chunk migration at a time and the copy is slow and resource-heavy, so the first choice still deserves care.
When to use it
Shard when one server's limits are the bottleneck you've named: a query rate that exhausts its CPU, or a working set larger than its RAM that turns into disk I/O. Our advice: when the problem is read volume, reach for a bigger machine or read replicas first, and shard when writes or total data outgrow one node.
Start from the query that runs most often and put its lookup value in the key. Take a chat app where you decide that "load this conversation's recent messages" is the dominant read: that argues for the chat ID as the partition key with message time as the ordering column, so one conversation is one partition. A second access, "everything new for this person across all their conversations", then becomes either a read across several partitions or a second copy of the data keyed by user, the choice described in Fan-out on write and on read. Say which you picked and what it costs: the storage and extra writes of a second copy, or the number of partitions a read touches.
When one key value must take more writes than one partition can handle (today's date in a log table, one very popular device), split it on purpose: append a suffix to the key so its writes spread over several partitions. A random suffix spreads writes evenly, but a read for that key then has to query all suffixes and merge the results; a suffix calculated from another attribute of the item can spread writes about as evenly and lets a reader work out which partition to ask. The diagnosis side is in Hot keys and hot partitions.
When not to
Don't shard a data set that fits in one partition with room to grow, where one partition's read and write capacity covers the load: there is no throttling to fix, and a sharded cluster adds infrastructure that needs planning, running and maintenance. Our advice: say out loud that you'd stay on one node with replicas until a limit you can name is reached.
Don't shard on a field with few distinct values (a status, a country, a boolean): the count of distinct values caps how many shards do any good. Be wary of sharding when most queries lack the key and can't be rewritten to carry it; without another index to route by, each one becomes a broadcast to all shards, and adding shards doesn't scale it linearly. The exception is heavy aggregation over most of the data, where running the same query in parallel on all shards is the point. And don't pick hashing when the main read is a range over the key (a day of events, a page of a conversation): hashing scatters neighbors, so a range strategy, or a composite key with the range in the ordering column, fits better.
Compared
| Strategy | How a key finds its shard | Pick it when | Pick something else when |
|---|---|---|---|
Lookup (directory) | A map from the key, or from a virtual shard, to a server | You want control: tenants pinned to a shard, data moved one tenant at a time, remaps without code changes | Requests can't afford the extra lookup, or there is no tenant-like grouping to pin; hash instead |
Range | Neighboring key values share a shard, stored in key order | The common read is a range (a month of orders, a page of messages) that should hit one shard | Activity sits on adjacent keys, such as the newest ones, so one shard runs hot and rebalancing is hard; hash the key |
Hash | A hash of the key picks the shard, with no map to keep | Writes must spread evenly, above all for keys that climb over time | Range scans over the key dominate, since they now touch many shards; or you add shards often and lack consistent hashing |
Geographic | A region attribute picks a shard hosted in that region | Data residency rules apply, or users in one region need low latency | Most users sit in one region, so its shard carries most of the load; combine it with hash or lookup inside the region |
In a picture
How to say it in an interview
The common mistake
Quiz
Questions that use this
Hear it in
Related
Sources
- MongoDB Manual: Choose a Shard Key
- MongoDB Manual: Sharding Introduction and Architecture
- Amazon DynamoDB Developer Guide: Designing partition keys to distribute your workload
- Amazon DynamoDB Developer Guide: Using write sharding to distribute workloads evenly
- Apache Cassandra 4.0 documentation: CQL Data Definition
- Azure Architecture Center: Sharding pattern