Delivery semantics: at most, at least, exactly once
The promise a message path makes when an acknowledgment goes missing: drop the message, send it again, or make the repeat harmless. A strong design answer names the promise at each hop and what absorbs a duplicate.
What it is
Delivery semantics describe what a messaging path promises when a failure interrupts a handoff. Kafka's design documentation sets out the three options: with at most once, a message may be lost but isn't redelivered; with at least once, a message isn't lost but may be redelivered; with exactly once, each message is delivered a single time. The same page warns that many claims of exactly once need their fine print read. The failure that separates the first two is ordinary: the sender sent, the receiver may have taken the message, and the acknowledgment didn't come back. If the sender retries, the receiver may see the message twice. If it doesn't, the message may be gone.
Two hops, two decisions. The first hop is producer to broker. In Kafka, a producer that expects a commit response and doesn't get one resends, which is at least once; since version 0.11.0.0 an idempotent option lets the broker assign the producer an ID and drop resends by a per-message sequence number, so a retry doesn't add a second copy to the log. In RabbitMQ 4.4, publisher confirms address the loss side rather than the duplicate side: the broker sends an ack once all the queues the message routed to have accepted it, and for a persistent message on a durable queue that means it reached disk. Without confirms, a message that the broker had in memory when it restarted is simply gone, and the publisher has no signal to resend it. A publisher that resends after a missing confirm can still produce a duplicate; the confirm says what was accepted, not which resend to drop.
The second hop is broker to consumer, and here the consumer chooses by when it acknowledges. Kafka's design documentation describes a consumer that saves its position before processing: a crash after the save loses whatever wasn't processed, which is at most once. A consumer that saves its position after processing can crash between the two, and the next instance reprocesses the last few messages, which is at least once. RabbitMQ offers the same choice as automatic acknowledgment (no ack needed) or manual, and its reliability guide states that using acks gives at least once while skipping them gives at most once. Amazon SQS does it with a visibility timeout: a received message stays in the queue but is hidden, for 30 seconds by default; if the consumer hasn't deleted it when the timeout ends, it becomes visible again for another consumer.
| Product feature | What it deduplicates | Where the promise stops |
|---|---|---|
Kafka idempotent producer (4.1 configs) | Producer retries, by producer ID and sequence number; needs | The write to the log; without idempotence, retries after broker failures may write duplicates |
Amazon SQS FIFO queue | Messages with the same deduplication ID, or the same body hash with content-based dedup turned on | A 5-minute deduplication interval; the same ID sent later is a new message |
Google Pub/Sub exactly-once delivery | Repeat deliveries of a message acked within its deadline, keyed on a Pub/Sub-assigned unique message ID; a subscriber still keeps its own progress record for acks that fail | One cloud region, pull subscriptions; push and export subscriptions don't support it; off by default |
What exactly once means in practice. Each feature in the table works inside a boundary, and the promise ends at that boundary. The same Kafka page says that exactly once into another system requires that system's cooperation, with the offset as the handle that makes it workable. Pub/Sub's guidance for subscribers says the same from the other side: acknowledge within the deadline, keep a record of how far processing got until the ack succeeds, and use that record to avoid repeating work when an ack fails. So the end-to-end version is at least once plus something on the consumer that makes a repeat harmless: an upsert keyed on the message, a stored idempotency key, or writing the result and the consumer position in one transaction.
When to use it
At least once is the default for work you can't afford to drop: orders, payments, notifications, anything with a side effect someone would miss. Kafka runs this way out of the box, and Amazon SQS standard queues promise it too, with the note that retries or network delays can deliver a message more than once and that handlers should be idempotent. The pairing is the point: at least once on the wire, plus a handler that does the right thing on the second copy. Idempotency keys cover the handler side.
At most once fits data where a missing sample costs less than a duplicate or a wait: metrics, presence pings, a typing indicator, a log line that will be superseded by the next one. The same design page gives the recipe: turn off producer retries and commit the consumer offset before processing a batch.
A product's exactly-once feature is worth its cost when the duplicate would be expensive to absorb downstream and the feature's boundary matches your problem. Amazon SQS FIFO queues are meant for cases where order matters or duplicates can't be tolerated, and the price is throughput: 300 API calls per second per action without batching, up to 3,000 messages per second with batches of ten, and more with high-throughput mode, which spreads the work across message groups while each group keeps its strict order. Kafka's transactional producer suits a pipeline that reads from one topic and writes to another, because the dedup and the transaction both live inside Kafka. For a pipeline that leaves the broker, treat the feature as one hop and keep the consumer idempotent.
When not to
Don't claim exactly once across a path whose last step leaves your system. Sending an email, charging a card through a third party or calling a partner API can succeed and then fail to report back, and no broker setting reaches that far: the SQS deduplication interval is 5 minutes at the queue, and Pub/Sub's feature covers redelivery within one region. The honest answer is at least once with a stored record per message ID that the handler checks before calling out again.
Don't pay for a dedup feature when the handler is already idempotent and order doesn't matter. The same Kafka design page notes that when messages carry a primary key, receiving one twice just overwrites a record with the same copy. If each consumer write is an upsert keyed that way and the events can be applied in any order, a standard queue with at least once does the job, and a FIFO queue's throughput limit buys you nothing you use. If order does matter, the FIFO queue earns its place for ordering, not for duplicates.
Don't reach for the heaviest mechanism either. RabbitMQ's documentation describes AMQP transactions as the older way to avoid loss on publish and says they cut throughput by a factor of about 250 compared with publisher confirms, which were added to replace them for that purpose. A confirm tells the publisher what the broker accepted, so it knows what to resend; a sequence number or a dedup ID lets the broker drop that resend. Together they are usually the lighter tool. And at most once isn't a shortcut for anything you'd have to apologize for losing; its place is data that the next message replaces anyway.
Compared
| Promise | On a lost ack, the sender | Failure you accept | When it's the right choice |
|---|---|---|---|
At most once | Gives up, or acked before the work began | A message is lost | Metrics, presence, anything the next message supersedes; the cheapest and fastest option |
At least once | Retries until acked; the receiver acks after the work | A message arrives twice | Anything you can't drop, when the handler can be made idempotent; the default in Kafka and in SQS standard queues |
Exactly once at one hop | Retries; the broker drops the repeat by sequence number or dedup ID, or a transaction commits the batch as a whole | Duplicates outside the window or region, and consumer crashes after the side effect | The duplicate is expensive downstream and the boundary matches: SQS FIFO within 5 minutes, Kafka inside its own topics, Pub/Sub within a region |
Exactly once end to end | Retries; the consumer upserts by key, or commits its result and its position in one transaction | Either a dedup store with a lookup per message, or a transaction that spans the result and the position | Money, inventory or anything where a repeat is visible to a user; builds on at least once rather than replacing it |
In a picture
Step 1 of 4
A message and an ack
A producer sends a message and waits for the broker's acknowledgment. In the happy path the ack arrives and nothing more happens.
How to say it in an interview
The common mistake
Quiz
Questions that use this
Hear it in
Related
Sources
- Apache Kafka 4.3 documentation: Design, Message Delivery Semantics
- Confluent documentation: Kafka message delivery semantics
- Apache Kafka 4.1 documentation: Producer Configs
- RabbitMQ documentation: Consumer Acknowledgments and Publisher Confirms
- RabbitMQ documentation: Consumers
- RabbitMQ documentation: Reliability Guide
- Amazon SQS Developer Guide: Amazon SQS queue types
- Amazon SQS Developer Guide: Amazon SQS FIFO queues
- Amazon SQS Developer Guide: Amazon SQS FIFO queue key terms
- Amazon SQS Developer Guide: Amazon SQS visibility timeout
- Google Cloud Pub/Sub documentation: Exactly-once delivery