Skip to content

Queues and logs

A queue hands each message to one worker and discards it once acknowledged; a log keeps records in order and lets each reader hold its own position. The choice decides fan-out, replay and how consumers scale.

On this page

What it is

A queue is a broker that holds a message until a consumer has processed it and said so. When a consumer receives a message from Amazon SQS, the message stays in the queue but becomes invisible to other consumers for a visibility timeout, 30 seconds by default; the consumer processes it and deletes it, and if it hasn't deleted it when the timeout runs out, the message becomes visible again for another consumer to pick up. RabbitMQ's classic queues work the same way with acknowledgments: a positive ack tells the broker the message was processed and can be discarded, and an unacknowledged delivery is requeued when the consumer's channel or connection closes. The result is competing consumers: each message is meant for one worker, and you add workers to go faster. Delivery is at least once, so a worker may occasionally see a message a second time and should be able to handle a duplicate.

A log is a sequence of records that grows at one end and is not edited. In Kafka 4.3's documentation, events in a topic are not deleted when they're consumed; instead a per-topic retention setting says how long to keep them, after which old events are discarded. A topic is split into partitions, events with the same key land in the same partition, and a consumer reads a partition's events in the order they were written. A consumer's position is an offset. Consumers that share a group id form a consumer group, each partition is read by one consumer in the group at a time, and a new group typically starts from either the earliest or the latest offset, then reads forward sequentially. Because the data stays, a second group can read the same topic without the first noticing.

IdeaQueueLog

Unit of progress

A message is done when its consumer deletes or acks it

A group commits an offset N, meaning N is the next record it intends to read

Who else sees the message

Nobody, once it's deleted

Any other group, until retention drops it

What a crash costs

The hidden message reappears after the timeout or when the connection closes

The consumer restarts from its last committed offset and re-reads from there

The two shapes are converging. RabbitMQ streams give a queue broker a log: a structure that is appended to and that consumers read repeatedly until records expire, described in its documentation as non-destructive consumer semantics, with a consumer choosing where to start through an x-stream-offset argument. Kafka went the other way: Queues for Kafka (KIP-932) shipped as a preview in 4.1 and is production-ready in 4.2. It adds share groups, where consumers cooperatively read a topic without one partition belonging to one consumer, with per-record acknowledgment and a count of delivery attempts, meant for records processed one at a time rather than as an ordered stream. The KIP notes that records in a share-partition may arrive out of order, particularly after a redelivery; a record's acquisition lock lasts 30 seconds by default and the delivery attempt limit defaults to 5.

When to use it

Reach for a queue when the items are independent jobs that one worker should handle once: send an email, resize an image, call a webhook. Throughput scales by adding consumers, with no partition count to plan up front. A queue also gives you per-message handling that a log typically lacks: the streams feature matrix lists message priority, TTL and dead-letter exchanges as supported on classic queues and unsupported on streams. When a job fails, redelivery is per message, and a message you don't want to process can be released early by setting its visibility timeout to zero.

Reach for a log when several systems need the same events. Large fan-out is the first use case the streams documentation lists, because with classic queues each subscriber needs its own bound queue. A log also fits when you'll want replay: a new service, or one recovering from a bug, can start at the earliest offset and rebuild its state from history, as long as retention still holds it. And it fits when order per key matters, such as all updates to one account arriving in sequence, since same-key events share a partition and are read in write order.

When not to

Don't put slow, uneven jobs on a log just because it's the bigger tool. A consumer group reads each partition sequentially, so if you process in order, a record that takes minutes delays the records behind it in that partition, and parallelism within a group is capped by the partition count you chose. Share groups lift that cap, but at the price of ordering within the group. If you need priority, TTL or a dead-letter path, those are classic queue features that streams don't offer.

Don't put shared events on a standard queue if a second reader is likely later: once a worker deletes a message, there is nothing left for the next system to read, so adding that reader means changing the producer or putting something that keeps events in front of the queue. When you need strict order on a queue, SQS FIFO queues preserve send order, deliver a message once and keep it unavailable until it's processed and deleted, with ordering scoped to message groups, but they carry a transactions-per-second limit; if you need both order and replay, that points at a log.

And sometimes neither: a single service with modest volume can keep a jobs table in the database it already has, with a status column and a worker that polls it. That costs nothing new to run, and it keeps the job and the data it changes in one transaction. Move to a broker when the polling load or the number of consumers makes the table the bottleneck.

Compared

PropertyQueueLogThe queue is right when

After processing

Deleted on ack or delete

Kept until retention expires

Nobody else will want the message and history is waste

Readers

One worker per message, any free one

Each group reads from its own position; within a group, one consumer per partition

Jobs are independent and a single system acts on them

Scaling consumers

Add workers

Add partitions; a group's active consumers are capped by the partition count

Load is bursty and job time is uneven

Ordering

FIFO queues order within a message group, at a throughput cost

In write order within a partition, so per key

Order doesn't matter, or matters per small group

Replay

Not possible after delete

Start a group at any offset that retention still holds

You won't need the history again

Failure handling

Hidden message reappears after a timeout; dead-letter after N tries

Group re-reads from its last committed offset; share groups add per-record ack

Each job needs its own retry and dead-letter path

In a picture

Order servicePartition 0,offsets 0..9

Step 1 of 4

Producers append

Writers add records to the end of a partition. Each record gets the next offset and the order is fixed from then on.

How to say it in an interview

The common mistake

Quiz

Six months after launch, a new team wants to build a report from the same events your worker has been processing. Which setup lets them do it without changes to the producer?
A topic has 6 partitions and a consumer group has 12 instances. How many instances receive records?
A worker crashes halfway through a message it received from a queue with a visibility timeout. What happens to the message?
You need all events for one account to be processed in order, and you have thousands of accounts. What gives you that on a log?

Questions that use this

Related

Sources

  1. Apache Kafka 4.3 documentation: Introduction
  2. Confluent documentation: Kafka consumer design, consumer groups and offsets
  3. Apache Kafka 4.4 documentation: Upgrading (notable changes in 4.1 and 4.2)
  4. KIP-932: Queues for Kafka
  5. Amazon SQS Developer Guide: Visibility timeout
  6. Amazon SQS Developer Guide: FIFO queues
  7. RabbitMQ documentation: Streams and superstreams
  8. RabbitMQ documentation: Consumer acknowledgments and publisher confirms
Sign in to report a mistake