How to design a distributed message queue
The first sections are free. The deep dives, level expectations and quiz are part of Pro.
On this page
- The problem
- Clarifying questions
- The requirements menu
- Non-functional requirements
- Estimates
- Core entities
- The API
- The high-level design
- What breaks first
- The deep dives
- Entity fields
- Replication and acks
- Consumers and delivery semantics
- Delayed messages
- Variants
- What each level is expected to show
- The 30-second answer
- Quiz
The problem
A distributed message queue takes messages from producers, appends each one to a partition of a topic, and lets consumers read them at their own pace, once or many times, for two weeks. The design turns on treating each partition as a replicated append-only log: order, durability, parallelism and retention all follow from that one choice. So does the main danger, because a single busy key sends all of its traffic to one partition and the broker that leads it.
Clarifying questions
| Ask | Why it matters | A typical answer |
|---|---|---|
How big are messages, and how fast do they arrive? | Size times rate is the write bandwidth a broker must absorb, and it decides whether one machine can lead a busy partition or whether the topic must be split further. | Messages of a few kilobytes, with each broker taking up to about a million messages a second. Keep individual messages under 1 MB. |
How long do we keep messages, and may a consumer read one again? | Retention sets the disk the cluster must hold, and re-reading rules out a queue that deletes on delivery in favor of a log consumers move through by position. | Two weeks, then truncated. A message can be read once or many times, so consumers keep their own position rather than the broker deleting what was read. |
What order do consumers need? | Order across the entire topic means putting all writes into one sequence, which caps throughput at what that one sequence takes. Order per key lets the topic split into partitions that run in parallel. | Order within a partition, where messages with the same key land. No order across the topic. |
Which delivery semantics do the consumers need? | At most once, at least once and exactly once differ in when the consumer records its position and in what the producer and broker must do about retries. Each step costs latency or complexity. | At least once at minimum, configurable per use case. Say what each setting costs when you pick it. |
When a broker dies right after acknowledging a message, may that message be lost? | The answer decides how many copies each partition keeps, and when the producer is told its write succeeded. | No. An acknowledged message should survive the loss of a broker, so partitions are copied across machines; three copies is common. |
Do producers need to delay a message, or filter what consumers see? | Delayed delivery needs a place for a message to wait inside the broker. Filtering by expression inside the broker is a different product. | Delayed delivery, yes: a payment check that runs 30 minutes after an order is a wanted case. Filtering inside the broker is out of scope. |
Non-functional requirements
| Requirement | What it forces |
|---|---|
Two-week retention, durable on disk and copied across nodes | Each partition is an append-only log on disk, split into segments so expiry deletes whole files. Each partition has copies on other brokers, and the producer is answered when enough copies hold the message, which is the first deep dive. |
High throughput or low latency, chosen per use case | Producers batch in memory before sending, and consumers fetch large batches; a producer that needs latency sends smaller batches sooner. The acknowledgment setting is per producer so each team picks its own point between fast and safe. |
Scales with a sudden surge of messages | Topics split into partitions spread across brokers, so adding brokers adds write capacity. Consumers pull at their own pace, so a surge piles up on disk in the log rather than in consumer memory, and the two-week retention gives them time to catch up. |
At-least-once delivery at minimum | A consumer records its position after processing a batch, not before, so after a crash the new owner reads that batch again. Consumers must tolerate seeing a message twice, which shapes each downstream side effect. |
Estimates
| Quantity | Working | Result | What it sizes |
|---|---|---|---|
Write bandwidth at the top rate | Assume 2 KB a message, within the stated few kilobytes. 1 million messages a second × 2 KB = 2 GB/s, about 16 Gbit/s; with 3 copies, about 6 GB/s of disk writes across the cluster | About 2 GB/s in from producers, about 6 GB/s written to disk once copied 3 times | The network and disk that a surge of a million messages a second needs, and why that traffic must spread across many partition leaders rather than one broker. |
Storage per day at the top rate | 2 GB/s × 86,400 seconds ≈ 172.8 TB a day, before copies | About 172.8 TB a day of new messages | How fast disk fills, which is the figure the retention period multiplies. |
Two weeks of messages, copied 3 times | 172.8 TB × 14 days ≈ 2.4 PB; × 3 copies ≈ 7.3 PB | About 2.4 PB raw, about 7.3 PB on disk with 3 copies | The total disk the cluster holds for two weeks of retention at a sustained million messages a second. |
Brokers needed to hold two weeks | 7.3 PB ÷ 1 TB a broker ≈ 7,300 brokers; at 1 KB a message, half that | About 7,300 brokers of 1 TB each at a sustained million messages a second | The broker count, which storage decides rather than throughput: that number is why you'd ask for far bigger disks per broker or name archiving to object storage as the escape, and why you should ask whether the million a second is a surge or the steady rate. |
Time to fill one broker's disk at its top rate | 1 TB ÷ 2 GB/s = 500 seconds, about 8 minutes | About 8 minutes | The gap between a broker's throughput and its storage: a broker that takes a million messages a second fills 1 TB in about 8 minutes, so the throughput figure does not size the cluster. |
Messages waiting if all messages are delayed 30 minutes | 30 minutes = 1,800 seconds; 1 million a second × 1,800 ≈ 1.8 billion messages; × 2 KB ≈ 3.6 TB | About 1.8 billion waiting messages, about 3.6 TB | Why delayed messages wait on disk in their own topics, with the timer holding references rather than the messages, and not in broker memory. |
Backlog during a rebalance pause | Assume a 10 second pause while partitions are reassigned; 1 million a second × 10 seconds = 10 million messages ≈ 20 GB at 2 KB each | About 10 million messages, about 20 GB to catch up on | The cost of a rebalance that stops the whole consumer group, and the argument for handing over just the partitions that move. |
Core entities
| Entity | What it holds |
|---|---|
Topic | A named stream of messages that producers write to and consumers subscribe to, split into partitions. |
Partition | An append-only log in segments, with one leader broker and follower copies; the unit of order, replication and parallelism. |
Message | One record in a partition, addressed by its offset. |
Consumer group | A set of consumers sharing a topic's partitions, each partition owned by one member, with a committed offset per partition. |
Broker | A machine that leads some partitions and follows others, taking up to about a million messages a second and holding about 1 TB. |
The API
| Endpoint | Notes |
|---|---|
| The producer hashes the key to pick the partition and sends to that partition's leader, batching messages in memory first. |
| Returns messages from the offset up to |
| Records the next offset to read for this group and partition, written to an internal offsets topic by the group coordinator. Call it after processing a batch for at least once; before, for at most once. |
| Finds the group's coordinator from a hash of the group id, joins, and gets back the partitions this member owns. Heartbeats keep the membership; a missed heartbeat window removes the member and triggers a rebalance. |
The high-level design
Step 1 of 5
One broker, an append-only log per partition
Start with the simplest version that fits the API. A producer sends a message to a broker, the broker appends it to the log for that partition and returns the offset. A consumer fetches from an offset and moves forward. The log is split into segment files; the newest segment takes writes, and expiry after two weeks deletes the oldest whole file instead of touching individual messages. Each topic has several partitions, and the key's hash picks one, so order holds for one key without a single global order. Sequential appends and sequential reads are what we'd rely on for one machine to take a high rate.
What breaks first
The deep dives
Replication and acks. The broker that took a message dies just after the producer was told the write succeeded, before any other broker had a copy. How do you make sure that message is still there, and what does the producer pay for that?
Consumers and delivery semantics. A consumer finishes processing a batch and crashes before it records that it did. What happens to those messages, how do the other consumers learn the partition is free, and which delivery promise does your commit order give?
Delayed messages. An order is placed and its payment check must run 30 minutes later. Where does that message wait, what releases it on time, and what happens if the broker holding it restarts in between?
Concepts this guide uses
Sources
- Confluent documentation: Kafka replication and committed messages
- Confluent documentation: Kafka message delivery guarantees
- Confluent Developer: Kafka internal architecture, consumer group protocol
- Confluent blog: exactly-once semantics in Apache Kafka (idempotent producer and transactions)
- Confluent blog: Apache Kafka, purgatory, and hierarchical timing wheels
- Apache RocketMQ documentation: Delayed message sending