Skip to content

How to design a distributed message queue

The first sections are free. The deep dives, level expectations and quiz are part of Pro.

On this page
Asked at
Practice this question

The problem

A distributed message queue takes messages from producers, appends each one to a partition of a topic, and lets consumers read them at their own pace, once or many times, for two weeks. The design turns on treating each partition as a replicated append-only log: order, durability, parallelism and retention all follow from that one choice. So does the main danger, because a single busy key sends all of its traffic to one partition and the broker that leads it.

Clarifying questions

AskWhy it mattersA typical answer

How big are messages, and how fast do they arrive?

Size times rate is the write bandwidth a broker must absorb, and it decides whether one machine can lead a busy partition or whether the topic must be split further.

Messages of a few kilobytes, with each broker taking up to about a million messages a second. Keep individual messages under 1 MB.

How long do we keep messages, and may a consumer read one again?

Retention sets the disk the cluster must hold, and re-reading rules out a queue that deletes on delivery in favor of a log consumers move through by position.

Two weeks, then truncated. A message can be read once or many times, so consumers keep their own position rather than the broker deleting what was read.

What order do consumers need?

Order across the entire topic means putting all writes into one sequence, which caps throughput at what that one sequence takes. Order per key lets the topic split into partitions that run in parallel.

Order within a partition, where messages with the same key land. No order across the topic.

Which delivery semantics do the consumers need?

At most once, at least once and exactly once differ in when the consumer records its position and in what the producer and broker must do about retries. Each step costs latency or complexity.

At least once at minimum, configurable per use case. Say what each setting costs when you pick it.

When a broker dies right after acknowledging a message, may that message be lost?

The answer decides how many copies each partition keeps, and when the producer is told its write succeeded.

No. An acknowledged message should survive the loss of a broker, so partitions are copied across machines; three copies is common.

Do producers need to delay a message, or filter what consumers see?

Delayed delivery needs a place for a message to wait inside the broker. Filtering by expression inside the broker is a different product.

Delayed delivery, yes: a payment check that runs 30 minutes after an order is a wanted case. Filtering inside the broker is out of scope.

The requirements menu

RequirementScopeWhy

Producers send messages and consumers read them

Core

The product. The rest refines how a message gets from a producer onto disk and from disk to a consumer.

A message can be read once or many times

Core

Several teams read the same topic, and a team may rewind after a bug. This is what makes it a log consumers move through, not a queue that deletes on delivery.

Order is kept within a partition

Core

Events about one key, say one order, must be seen in sequence. Promising more than this, order across a topic, means putting all writes to the topic into one sequence, which gives up parallelism.

Messages are kept for two weeks, then truncated

Core

Retention decides the storage the cluster holds and makes deleting cheap: drop whole old segments rather than individual messages.

Delivery is configurable: at most once, at least once or exactly once

Core

Different consumers can afford different things: metrics can lose a batch, payments must not run twice. The design has to offer each one and say what it costs.

Deliver a message after a chosen delay

Optional

Worth taking when asked, since the prompt's payment check, which runs a set time after an order, needs it. It needs a holding place inside the broker and a timer, which is one of the deep dives.

A dead-letter topic for messages a consumer keeps failing on

Optional

Useful in production, because one bad message can otherwise hold up a partition behind it while the consumer keeps retrying it. Name it; the mechanism is a consumer that publishes to another topic after a retry limit, so it needs little design time.

Per-client quotas on produce and fetch rate

Optional

Many teams share the cluster, so one runaway producer can starve the rest. Take it if the interviewer asks about multi-tenancy; otherwise mention it and move on.

Keep the latest message per key instead of truncating by age

Optional

Compaction turns a topic into a changelog that holds current state. Tempting, but it adds a background cleaning job to the storage layer; leave it unless the use case is a changelog.

Let a consumer start from a point in time rather than an offset

Optional

Handy for replaying since an incident. In this design it would need an index from time to offset per segment, so mention it and park it.

Filtering by expressions inside the broker

Out of scope

Server-side filtering means the broker inspects message contents instead of copying bytes, which costs it throughput. Consumers filter after fetching.

AMQP compatibility

Out of scope

A protocol layer with its own routing model. It adds nothing to the storage and replication questions this round is about.

Archiving old messages to object storage

Out of scope

Archiving to object storage is the escape from the broker count the estimate below produces. Name it as a way to hold two weeks without thousands of disks, then leave it.

Non-functional requirements

RequirementWhat it forces

Two-week retention, durable on disk and copied across nodes

Each partition is an append-only log on disk, split into segments so expiry deletes whole files. Each partition has copies on other brokers, and the producer is answered when enough copies hold the message, which is the first deep dive.

High throughput or low latency, chosen per use case

Producers batch in memory before sending, and consumers fetch large batches; a producer that needs latency sends smaller batches sooner. The acknowledgment setting is per producer so each team picks its own point between fast and safe.

Scales with a sudden surge of messages

Topics split into partitions spread across brokers, so adding brokers adds write capacity. Consumers pull at their own pace, so a surge piles up on disk in the log rather than in consumer memory, and the two-week retention gives them time to catch up.

At-least-once delivery at minimum

A consumer records its position after processing a batch, not before, so after a crash the new owner reads that batch again. Consumers must tolerate seeing a message twice, which shapes each downstream side effect.

Estimates

QuantityWorkingResultWhat it sizes

Write bandwidth at the top rate

Assume 2 KB a message, within the stated few kilobytes. 1 million messages a second × 2 KB = 2 GB/s, about 16 Gbit/s; with 3 copies, about 6 GB/s of disk writes across the cluster

About 2 GB/s in from producers, about 6 GB/s written to disk once copied 3 times

The network and disk that a surge of a million messages a second needs, and why that traffic must spread across many partition leaders rather than one broker.

Storage per day at the top rate

2 GB/s × 86,400 seconds ≈ 172.8 TB a day, before copies

About 172.8 TB a day of new messages

How fast disk fills, which is the figure the retention period multiplies.

Two weeks of messages, copied 3 times

172.8 TB × 14 days ≈ 2.4 PB; × 3 copies ≈ 7.3 PB

About 2.4 PB raw, about 7.3 PB on disk with 3 copies

The total disk the cluster holds for two weeks of retention at a sustained million messages a second.

Brokers needed to hold two weeks

7.3 PB ÷ 1 TB a broker ≈ 7,300 brokers; at 1 KB a message, half that

About 7,300 brokers of 1 TB each at a sustained million messages a second

The broker count, which storage decides rather than throughput: that number is why you'd ask for far bigger disks per broker or name archiving to object storage as the escape, and why you should ask whether the million a second is a surge or the steady rate.

Time to fill one broker's disk at its top rate

1 TB ÷ 2 GB/s = 500 seconds, about 8 minutes

About 8 minutes

The gap between a broker's throughput and its storage: a broker that takes a million messages a second fills 1 TB in about 8 minutes, so the throughput figure does not size the cluster.

Messages waiting if all messages are delayed 30 minutes

30 minutes = 1,800 seconds; 1 million a second × 1,800 ≈ 1.8 billion messages; × 2 KB ≈ 3.6 TB

About 1.8 billion waiting messages, about 3.6 TB

Why delayed messages wait on disk in their own topics, with the timer holding references rather than the messages, and not in broker memory.

Backlog during a rebalance pause

Assume a 10 second pause while partitions are reassigned; 1 million a second × 10 seconds = 10 million messages ≈ 20 GB at 2 KB each

About 10 million messages, about 20 GB to catch up on

The cost of a rebalance that stops the whole consumer group, and the argument for handing over just the partitions that move.

Core entities

EntityWhat it holds

Topic

A named stream of messages that producers write to and consumers subscribe to, split into partitions.

Partition

An append-only log in segments, with one leader broker and follower copies; the unit of order, replication and parallelism.

Message

One record in a partition, addressed by its offset.

Consumer group

A set of consumers sharing a topic's partitions, each partition owned by one member, with a committed offset per partition.

Broker

A machine that leads some partitions and follows others, taking up to about a million messages a second and holding about 1 TB.

The API

EndpointNotes

produce(topic, key, value, acks) → { partition, offset }

The producer hashes the key to pick the partition and sends to that partition's leader, batching messages in memory first. acks is 0, 1 or all: 0 waits for no reply, 1 for the leader's append, all for the in-sync copies. In this design a missing key means round-robin across partitions, so order is not promised for keyless messages.

fetch(topic, partition, offset, max_bytes) → messages

Returns messages from the offset up to max_bytes, and no further than the last offset all in-sync copies hold. In this design a fetch with nothing new waits a short while before answering, so idle consumers do not spin.

commit(group, topic, partition, offset)

Records the next offset to read for this group and partition, written to an internal offsets topic by the group coordinator. Call it after processing a batch for at least once; before, for at most once.

join_group(group, topics) → assigned partitions

Finds the group's coordinator from a hash of the group id, joins, and gets back the partitions this member owns. Heartbeats keep the membership; a missed heartbeat window removes the member and triggers a rebalance.

The high-level design

producefetchProducerPartition leadersConsumer

Step 1 of 5

One broker, an append-only log per partition

Start with the simplest version that fits the API. A producer sends a message to a broker, the broker appends it to the log for that partition and returns the offset. A consumer fetches from an offset and moves forward. The log is split into segment files; the newest segment takes writes, and expiry after two weeks deletes the oldest whole file instead of touching individual messages. Each topic has several partitions, and the key's hash picks one, so order holds for one key without a single global order. Sequential appends and sequential reads are what we'd rely on for one machine to take a high rate.

delayeddueleadersproducefetchcopyheartbeatcommitProducerDelay topicsTime wheelPartition leadersCoordinationserviceConsumerIn-sync followersGroup coordinatorCommittedoffsets
The whole design

What breaks first

The deep dives

Replication and acks. The broker that took a message dies just after the producer was told the write succeeded, before any other broker had a copy. How do you make sure that message is still there, and what does the producer pay for that?

A concept that helps here: Replication and read replicasA concept that helps here: Leader election and failoverA concept that helps here: Quorums and tunable consistency

Consumers and delivery semantics. A consumer finishes processing a batch and crashes before it records that it did. What happens to those messages, how do the other consumers learn the partition is free, and which delivery promise does your commit order give?

A concept that helps here: Delivery semantics: at most, at least, exactly onceA concept that helps here: Idempotency keysA concept that helps here: Heartbeats and presence

Delayed messages. An order is placed and its payment check must run 30 minutes later. Where does that message wait, what releases it on time, and what happens if the broker holding it restarts in between?

A concept that helps here: Job scheduling and delayed workA concept that helps here: Idempotency keys

Concepts this guide uses

Sources

  1. Confluent documentation: Kafka replication and committed messages
  2. Confluent documentation: Kafka message delivery guarantees
  3. Confluent Developer: Kafka internal architecture, consumer group protocol
  4. Confluent blog: exactly-once semantics in Apache Kafka (idempotent producer and transactions)
  5. Confluent blog: Apache Kafka, purgatory, and hierarchical timing wheels
  6. Apache RocketMQ documentation: Delayed message sending
Sign in to report a mistake