Skip to content

Retries and dead-letter queues

A retry repeats a failed call, usually after a growing, randomized wait; a dead-letter queue parks a message whose retries ran out so the queue keeps flowing and the message can be inspected and replayed.

On this page

What it is

A retry sends the same request again after it failed or timed out. It works because most failures in a distributed system are partial (some requests fail) or transient (requests fail for a short while), and the same request a moment later often goes through. A timeout is what turns a hanging call into a failure you can act on: without one the client holds threads, connections and memory while it waits, and a slow dependency drags its callers down with it.

A retry is not free. Each one asks the dependency to spend more of its capacity on your request, and when the dependency is failing because it is overloaded, a wave of retries can keep it down after the original cause is gone. Three habits bound that cost. Backoff: wait longer after each attempt, with the wait growing exponentially, and with a cap so it doesn't grow without end. Jitter: add randomness to each wait, because clients that failed at the same moment would otherwise retry at the same moment and collide again. A limit on attempts: after a few tries, give up and return the failure to the caller, which usually has its own timeout anyway.

Some failures don't deserve a retry. HTTP's split between client errors (4xx) and server errors (5xx) is the usual starting point: a request rejected because it is wrong, such as a validation failure, won't pass by being resent, while a server error may pass next time. A useful framing names three responses: cancel when the fault isn't transient, retry at once for a rare one-off such as a corrupted packet, and retry after a delay for the common busy or connectivity faults. One caveat: eventual consistency blurs the line, since a request rejected as invalid one moment can succeed once state has propagated.

A dead-letter queue (DLQ) is a second queue where a broker parks a message that a consumer kept failing to process. The consumer receives the message, fails, and doesn't acknowledge it; the broker makes it available again; after roughly a configured number of receives (brokers count differently, and some dead-letter for other reasons too, such as an expired TTL) the broker moves it aside instead of delivering it once more. The message is handed to the DLQ rather than discarded, and the main queue no longer spends time on it. A message that fails on each attempt because of its content rather than a passing fault is the case to picture: the DLQ isolates it so you can work out why it didn't process.

BrokerWhat sends a message to the DLQWhat the DLQ copy carries

Amazon SQS

A redrive policy with maxReceiveCount: the number of times a consumer may receive the message before the broker moves it. Set to 1, a single failed receive moves it.

The original message, with its original enqueue timestamp on a standard queue

Google Pub/Sub

A dead-letter topic with a maximum delivery attempts value between 5 and 100. The count is approximate and may reset to zero, so a message can be delivered more times than configured.

The original wrapped in a new message, with attributes naming the source subscription

RabbitMQ 4.4

Any of four events: a consumer rejects it without requeue, its per-message TTL expires, its queue exceeds a length limit, or a quorum queue returns it more times than its delivery-limit.

The message with its dead-lettering history, most recent first (an x-death header over AMQP 0.9.1)

The DLQ is where the work starts, not where it ends. The point of keeping the message is to read it next to the consumer's logs, work out whether the consumer had too little time, the message was bad or a dependency was down, fix that, and then move the messages back to the source queue, which SQS calls a redrive. Two details matter for that. On a standard queue there, a moved message keeps its original enqueue time, so its retention clock keeps running from when it was first sent, which is why the DLQ's retention period should be longer than the source queue's. And a DLQ nobody looks at is a slow leak, so put an alarm on messages arriving in it.

When to use it

Retry a call when the fault is expected to be short-lived and the same request could pass on the next attempt: a dropped connection, a busy dependency that is throttling, a timeout. Do it at a single layer of the stack. Amazon's engineers give the arithmetic: a call that goes five services deep with three tries at each layer multiplies the load on a failing database by 243, because the retries at each layer multiply. Google's SRE book makes the same point from the other side: retry at the layer directly above the one that rejected the request, and pass an "overloaded; don't retry" error upward so the layers above don't pile on.

Give retries a budget, not just a count. The SRE book describes two limits used at Google: at most three attempts per request, and a per-client limit where a request is retried as long as retries stay below 10% of that client's traffic. In their worst case the first limit on its own still lets load grow to nearly 3x, and the 10% ratio brings it to about 1.1x. A related device is a token bucket of retries inside the client, which AWS added to its SDK in 2016: calls may retry while tokens remain, and once they run out retries continue at a fixed rate. Both are alternatives to a circuit breaker that stops all calls, at the price of modal behavior that is hard to test and a slower return to normal.

Shape the policy to the caller. For an interactive request a sensible policy is a small number of retries with short delays and then a message to the user; for a batch job, more attempts with exponentially growing delays are fine because nobody is waiting on the screen. Log the early failures at an informational level and the final failure as an error, so operators aren't paged for faults a retry already fixed.

Add a dead-letter queue when a consumer works from a shared queue and a message can fail for a reason a retry won't fix: a payload the code can't parse, a reference to a record that was deleted, a bug in the handler for one rare case. If nothing sets that message aside, it will either keep coming back to the consumer, eating capacity on each pass, or be dropped and gone. With a DLQ, the bad message is set aside after its attempts run out, the rest of the queue keeps flowing, and you keep the evidence you need to debug it and the ability to replay it after the fix.

When not to

Don't retry a fault that won't clear on its own. The Azure retry pattern lists the cases: a long-lasting outage, where repeated attempts waste time and keep the user waiting; failures that come from the application's own business logic rather than a transient fault; and frequent busy errors, which are a sign the service needs more capacity, not a longer retry loop. A validation error is the everyday example: resending the same bad request is wasted work.

Don't retry an operation with side effects unless it is idempotent. A timeout tells you the response didn't arrive, not that the work didn't happen; the server may have charged the card or created the order and failed to reply. Amazon's view is that an API with side effects isn't safe to retry unless it provides idempotency, for example through a client token as EC2's RunInstances does. If you can't make the handler idempotent, limit retries to the steps that have no side effects and surface the ambiguous result instead. Idempotency keys covers the usual mechanism.

Be careful with dead-letter queues where order matters. The SQS documentation advises against pairing a DLQ with a FIFO queue when the exact order of operations has to hold, because pulling one message aside changes the order of what follows. Likewise, a DLQ is the wrong tool when a whole dependency is down: moving thousands of healthy messages into it as they each exhaust their attempts leaves you with a mass redrive later. Our advice: when failures are correlated, slow or pause consumption instead, and let the DLQ catch the stragglers. Backpressure and load shedding covers that side.

Check what the DLQ itself promises. In RabbitMQ 4.4, dead-lettering is a form of publishing, and by default the broker republishes without publisher confirms and removes the message from the source queue right after publishing, so in a cluster a dead-lettered message can be lost; quorum queues offer an at-least-once mode that turns confirms on. If the configured dead-letter exchange doesn't exist when a message needs it, the message is dropped without a trace. A DLQ is a safety net with holes you should know about, not a durable archive.

Compared

OptionWhat it doesWhen it's the right callWhat it costs

Retry at once, no delay

Resends immediately

A rare one-off fault such as a corrupted packet, where the dependency isn't the problem

Under overload, it adds load exactly when the dependency can least take it

Capped exponential backoff with jitter

Waits longer each time, randomized, up to a cap and a fixed number of attempts

A dependency that may be busy or briefly unreachable, where the fault is expected to clear on its own

Latency for the caller while it waits; still a fixed multiple of extra load

Retry budget (token bucket or ratio)

Lets retries happen while a budget lasts, then holds them to a trickle

Many clients share a dependency and a correlated failure could become a retry storm

More state in the client; a request may be refused a retry that would have worked

Circuit breaker

Stops all calls after an error threshold, then probes

You'd rather fail fast than spend capacity on a dependency that is clearly down

Switches between modes abruptly, which is hard to test, and can delay recovery

Dead-letter queue

Parks a message after its attempts run out

A consumer on a shared queue meets messages that fail for a reason a retry won't fix

Someone has to watch it, diagnose and redrive; order can change

Drop after the last attempt

Gives up and discards

The message is low-value and stale once late, such as a presence ping or a metric sample

The data is gone and so is the evidence

In a picture

okSource queueWorkerDelete message

Step 1 of 4

The happy path

A worker receives a message from the source queue, does the work, and deletes (acknowledges) the message. Nothing else is involved.

How to say it in an interview

The common mistake

Quiz

A downstream service answers three calls: a 400 for a malformed body, a 503 while it is overloaded, and a timeout. Which is the clearest case for not retrying?
Why add jitter to exponential backoff?
A request passes through five services before it reaches a database, and on failure each layer makes up to three tries in total (the first call plus two retries). When the database starts failing, roughly how much load does it see compared with normal?
Messages are landing in the dead-letter queue because the worker crashes on a field some producers leave empty. You fix the bug. What happens to the messages already in the dead-letter queue?

Questions that use this

Related

Sources

  1. Amazon Builders' Library: Timeouts, retries, and backoff with jitter
  2. Azure Architecture Center: Retry pattern
  3. Amazon SQS Developer Guide: Using dead-letter queues in Amazon SQS
  4. Google Cloud Pub/Sub documentation: Dead-letter topics
  5. RabbitMQ 4.4 documentation: Dead Letter Exchanges
  6. Google SRE book: Handling Overload
Sign in to report a mistake