Queues and logs
A queue hands each message to one worker and discards it once acknowledged; a log keeps records in order and lets each reader hold its own position. The choice decides fan-out, replay and how consumers scale.
What it is
A queue is a broker that holds a message until a consumer has processed it and said so. When a consumer receives a message from Amazon SQS, the message stays in the queue but becomes invisible to other consumers for a visibility timeout, 30 seconds by default; the consumer processes it and deletes it, and if it hasn't deleted it when the timeout runs out, the message becomes visible again for another consumer to pick up. RabbitMQ's classic queues work the same way with acknowledgments: a positive ack tells the broker the message was processed and can be discarded, and an unacknowledged delivery is requeued when the consumer's channel or connection closes. The result is competing consumers: each message is meant for one worker, and you add workers to go faster. Delivery is at least once, so a worker may occasionally see a message a second time and should be able to handle a duplicate.
A log is a sequence of records that grows at one end and is not edited. In Kafka 4.3's documentation, events in a topic are not deleted when they're consumed; instead a per-topic retention setting says how long to keep them, after which old events are discarded. A topic is split into partitions, events with the same key land in the same partition, and a consumer reads a partition's events in the order they were written. A consumer's position is an offset. Consumers that share a group id form a consumer group, each partition is read by one consumer in the group at a time, and a new group typically starts from either the earliest or the latest offset, then reads forward sequentially. Because the data stays, a second group can read the same topic without the first noticing.
| Idea | Queue | Log |
|---|---|---|
Unit of progress | A message is done when its consumer deletes or acks it | A group commits an offset N, meaning N is the next record it intends to read |
Who else sees the message | Nobody, once it's deleted | Any other group, until retention drops it |
What a crash costs | The hidden message reappears after the timeout or when the connection closes | The consumer restarts from its last committed offset and re-reads from there |
The two shapes are converging. RabbitMQ streams give a queue broker a log: a structure that is appended to and that consumers read repeatedly until records expire, described in its documentation as non-destructive consumer semantics, with a consumer choosing where to start through an x-stream-offset argument. Kafka went the other way: Queues for Kafka (KIP-932) shipped as a preview in 4.1 and is production-ready in 4.2. It adds share groups, where consumers cooperatively read a topic without one partition belonging to one consumer, with per-record acknowledgment and a count of delivery attempts, meant for records processed one at a time rather than as an ordered stream. The KIP notes that records in a share-partition may arrive out of order, particularly after a redelivery; a record's acquisition lock lasts 30 seconds by default and the delivery attempt limit defaults to 5.
When to use it
Reach for a queue when the items are independent jobs that one worker should handle once: send an email, resize an image, call a webhook. Throughput scales by adding consumers, with no partition count to plan up front. A queue also gives you per-message handling that a log typically lacks: the streams feature matrix lists message priority, TTL and dead-letter exchanges as supported on classic queues and unsupported on streams. When a job fails, redelivery is per message, and a message you don't want to process can be released early by setting its visibility timeout to zero.
Reach for a log when several systems need the same events. Large fan-out is the first use case the streams documentation lists, because with classic queues each subscriber needs its own bound queue. A log also fits when you'll want replay: a new service, or one recovering from a bug, can start at the earliest offset and rebuild its state from history, as long as retention still holds it. And it fits when order per key matters, such as all updates to one account arriving in sequence, since same-key events share a partition and are read in write order.
When not to
Don't put slow, uneven jobs on a log just because it's the bigger tool. A consumer group reads each partition sequentially, so if you process in order, a record that takes minutes delays the records behind it in that partition, and parallelism within a group is capped by the partition count you chose. Share groups lift that cap, but at the price of ordering within the group. If you need priority, TTL or a dead-letter path, those are classic queue features that streams don't offer.
Don't put shared events on a standard queue if a second reader is likely later: once a worker deletes a message, there is nothing left for the next system to read, so adding that reader means changing the producer or putting something that keeps events in front of the queue. When you need strict order on a queue, SQS FIFO queues preserve send order, deliver a message once and keep it unavailable until it's processed and deleted, with ordering scoped to message groups, but they carry a transactions-per-second limit; if you need both order and replay, that points at a log.
And sometimes neither: a single service with modest volume can keep a jobs table in the database it already has, with a status column and a worker that polls it. That costs nothing new to run, and it keeps the job and the data it changes in one transaction. Move to a broker when the polling load or the number of consumers makes the table the bottleneck.
Compared
| Property | Queue | Log | The queue is right when |
|---|---|---|---|
After processing | Deleted on ack or delete | Kept until retention expires | Nobody else will want the message and history is waste |
Readers | One worker per message, any free one | Each group reads from its own position; within a group, one consumer per partition | Jobs are independent and a single system acts on them |
Scaling consumers | Add workers | Add partitions; a group's active consumers are capped by the partition count | Load is bursty and job time is uneven |
Ordering | FIFO queues order within a message group, at a throughput cost | In write order within a partition, so per key | Order doesn't matter, or matters per small group |
Replay | Not possible after delete | Start a group at any offset that retention still holds | You won't need the history again |
Failure handling | Hidden message reappears after a timeout; dead-letter after N tries | Group re-reads from its last committed offset; share groups add per-record ack | Each job needs its own retry and dead-letter path |
In a picture
Step 1 of 4
Producers append
Writers add records to the end of a partition. Each record gets the next offset and the order is fixed from then on.
How to say it in an interview
The common mistake
Quiz
Questions that use this
Related
Sources
- Apache Kafka 4.3 documentation: Introduction
- Confluent documentation: Kafka consumer design, consumer groups and offsets
- Apache Kafka 4.4 documentation: Upgrading (notable changes in 4.1 and 4.2)
- KIP-932: Queues for Kafka
- Amazon SQS Developer Guide: Visibility timeout
- Amazon SQS Developer Guide: FIFO queues
- RabbitMQ documentation: Streams and superstreams
- RabbitMQ documentation: Consumer acknowledgments and publisher confirms