Skip to content

Backpressure and load shedding

Two ways to survive a sender that outruns a receiver: backpressure makes the sender slow down, and load shedding rejects excess requests early and cheaply so the work you do accept still finishes in time.

On this page

What it is

Backpressure is what happens when data piles up behind a buffer because the receiving side is slower than the sending side, and it is also the name for the fix: a signal from the receiver that the sender honors by pausing or slowing down. Unix pipes and TCP sockets do this and call it flow control. In Node.js streams the signal is the return value of a writable stream's write() call, which tells the source whether it may keep sending.

Load shedding is the other response: as a server approaches overload it drops some share of the incoming traffic on purpose, so that it still completes the most useful work possible instead of running out of memory, failing its health checks or answering so slowly that the answers are useless. A simple form is a bounded queue in front of the worker threads that rejects new requests when it is full; another is returning an HTTP 503 once more than a set number of requests are in flight.

Why a server needs either one: overload amplifies itself. Throughput is the total number of requests arriving; goodput is the part answered without error and fast enough for the client to use. Once a server's median latency reaches the client timeout, half of its responses arrive after the client has given up, so a latency problem becomes an availability problem. Each timed-out request wastes the work already done on it, and clients often retry, which multiplies the offered load. In a deep call chain where each layer retries, the multiplication compounds at each hop. Shedding early keeps goodput roughly flat while offered load keeps climbing, until the cost of rejecting itself becomes the bottleneck.

MechanismHow it signalsWhere you meet it

Flow control

The receiver tells the sender how much it can take; the sender pauses

TCP, Unix pipes, stream libraries

Bounded queue

The queue refuses new work when full

The request queue in front of a worker pool

Fast rejection

A 503 response, ideally with Retry-After saying when to come back

Overloaded web servers and proxies

Latency-driven limit

Latency samples raise or lower the number of in-flight requests allowed

Envoy 1.24.1's adaptive concurrency filter

Deadline propagation

Each hop passes on the time remaining; work stops when it runs out

RPC frameworks, request queues

Graceful degradation

Cheaper, lower-quality answers instead of refusals

Search over a cache subset, simpler ranking

Queues are where overload hides. A queued request costs memory and adds waiting time. If the queue holds ten times as many requests as there are threads and each request takes 100 ms of worker time, a request that enters a full queue takes about 1.1 seconds, nearly all of it waiting. A queue that is small relative to the thread pool, about half its size or less, makes the server say no early when it can't keep up. Bursty traffic is the exception where a deeper queue, sized from the threads in use, the per-request time and the burst shape, can be worth it.

Size is not the whole story: bound the time a request may sit in a queue too, and throw it out when that bound passes, because by the time a long-queued request reaches a worker the client may have timed out. Record how long each dequeued request waited. In-memory queues also turn up in places you didn't put them, such as the executor inside a web framework and the operating system's socket buffers, so assume there are queues you haven't found and load-test past the breaking point to see how they behave. A spillover setting that fails fast instead of queuing excess is the safer default; Amazon's Application Load Balancer rejects excess traffic where the older Classic Load Balancer kept a surge queue.

What to drop first. The one request to protect is the load balancer's health check: miss it and the balancer stops sending traffic to that server, shrinking the fleet in the middle of a brownout. Beyond that, prioritize by who is asking and by sunk cost: a human's request over a crawler's, the call that finishes a multi-step job over the call that starts a new one, and page N of a paginated read over page one, since dropping late means throwing away everything already spent. Make these decisions as early in the stack as possible, in the service that first receives the request, rather than guessing at priorities deep inside.

Watch the clock. A request whose client has already given up is wasted work, so a server that notices the deadline has passed partway through can stop. The deadline is set once, high in the stack, and each hop passes on what remains: a 30-second deadline that spends 7 seconds in server A reaches server B with 23 seconds left. Measure durations with a monotonic clock, since wall clocks jump when they resync; and remember that the stopwatch may already be late, because a request can sit in a TCP buffer before the server reads it. Related to this, a backend that is known to be down should fail immediately rather than holding a worker until the deadline, or a small fraction of hanging requests can tie up most of the worker threads.

Adaptive limits replace a hand-tuned number with a control loop. Envoy 1.24.1's adaptive concurrency filter periodically measures an ideal round-trip time (minRTT) by allowing a very small number of outstanding requests to the upstream, then compares sampled latencies against it. It computes a gradient, (minRTT + B) / sampleRTT where B is a buffer that tolerates normal variance, and sets the new limit to gradient times the old limit plus a headroom pinned to the square root of the limit. As sampled latency rises the gradient shrinks, which pulls the limit down once the drop outweighs the headroom; while latency stays near minRTT the headroom keeps nudging the limit up. The documentation warns that 503 responses rise during the minRTT window, because the limit drops to 3 by default, and recommends retries to another host.

When to use it

Reach for backpressure when the sender can hold its data and wait: a file being compressed to disk, a worker pulling from a log at its own pace, a stage in a pipeline. The Node.js guide shows what the signal is worth: compressing a file while respecting the write() return value peaked at about 88 MB of memory, and ignoring it peaked at about 1.52 GB for the same job. That is the general shape of the win: the sender's pace is capped by the receiver's, so buffers stay bounded.

Reach for load shedding when the sender can't wait or the request loses its value as it ages. A user whose search has been queued for ten seconds has probably refreshed the page and sent another one, so answering the first is pointless; serving the newest requests first (last in, first out) or a controlled-delay queue discipline tends to shed the long-waiting requests that are least likely to still matter, and works best when deadlines travel with the request. Return 503 for the rejected ones, since that code is meant for temporary conditions and pairs with a Retry-After header carrying either a date or a number of seconds to wait. Don't let caches store the 503, or clients may keep seeing the error page after recovery.

Prefer graceful degradation over outright refusal when a cheaper answer still helps the user: searching a cached subset instead of the full index, or a faster, less accurate ranking. Shared services serving many clients are where selective shedding by client or by request importance pays off most.

When not to

Don't make shedding the first line of defense against load you can predict. Capacity and automatic scaling come first; shedding is for the surge, the lost capacity after a bad deploy, or the day clients switch from cheap cached reads to expensive misses. Degraded modes should trip rarely, and a mode that rarely runs is a mode you have little experience operating, so keep it simple, exercise it on purpose by running a few servers near overload, and build a quick way to turn it off, since complex degradation logic can itself push a server into degraded mode or into a feedback cycle at the wrong moment.

Don't drop a request when dropping is dearer than serving it. Dropping has a cost, and a stray log line or socket setting can make it much larger than expected; load-test far past the plateau to find out. In the rare case where rejecting is the more expensive path, slowing rejected requests down to at least the latency of a success, while they hold no application thread, beats a fast refusal.

Don't treat a per-client rate limit or a proxy's connection cap as the whole answer to overload. A per-client rate limit doesn't look at the service's health, so it may not stop a failure already under way, and a simple one tends to leave capacity idle. A max_conns style cap on a proxy does shed load, but it is hard to prioritize by request importance and the raw in-flight count can misjudge whether the service is in trouble, which is why it belongs as a last resort rather than the default guard. Early rejection at a proxy or firewall is the cheapest place to drop, but you lose visibility into what you dropped, so protect in layers and log enough at each one.

And don't count on backpressure across a boundary where the producer can't pause: a person clicking a button, a partner's webhook, a device that emits readings on a timer. The signal has nowhere useful to land, so the buffer just moves one hop upstream. Our advice: at that boundary, accept into a bounded, durable queue or shed; use backpressure between the stages you control behind it.

Compared

OptionWhat it doesPick it whenA simpler option is right when

Backpressure (flow control)

Receiver paces the sender; buffers stay bounded

Sender can pause and hold its data: pipelines, log consumers, file transfers

The sender can't pause, so the signal has no effect; shed or buffer durably instead

Bounded queue with fast reject

Refuses new work when the queue or in-flight count hits a fixed limit

Steady traffic and a known per-request cost; you want a cheap, predictable guard

Capacity or the cost per request varies, so a fixed number is too low at some times and too high at others

Latency-driven in-flight limit

Moves the in-flight cap up and down from latency samples

Upstream capacity shifts, per-request cost varies, and you can route all traffic through the filter

Some traffic bypasses the control point; a fixed limit is easier to reason about and debug

Deadline-based dropping

Discards requests whose client has already given up

Deep call chains where late answers are worthless

Single-hop services with short, uniform latency, where a bounded queue already catches it

Graceful degradation

Serves a cheaper answer instead of refusing

A partial or approximate answer still helps the user

There is no useful cheaper answer (a payment, a write), so a clear refusal is more honest

Rate limiting per client

Caps each caller's request rate

Fairness between tenants and abuse control

Overload comes from legitimate load the limit can't see, so it neither stops it nor uses idle capacity

In a picture

ClientsLoad balancerRequest queueWorker threadsDatabase

Step 1 of 3

The unprotected path

Requests flow through a load balancer into a request queue, onto worker threads, and to the database. With no bound on the queue, a surge makes it grow, waiting time passes the client timeout, and retries add to the pile.

How to say it in an interview

The common mistake

Quiz

A service's median response time has risen to exactly the client's timeout. Roughly what fraction of requests does the client see succeed?
An overloaded server must turn some requests away. Which one should it keep serving first?
A worker pool has a queue holding ten times as many requests as it has threads, and each request takes 100 ms of worker time. How long does a request take when it arrives to a full queue?
A batch job reads a large file, compresses it and writes it to a slower disk. Which approach keeps memory bounded?

Questions that use this

Related

Sources

  1. Amazon Builders' Library: Using load shedding to avoid overload
  2. Google SRE book: Addressing Cascading Failures
  3. Node.js Learn: Backpressuring in Streams
  4. MDN: 503 Service Unavailable
  5. MDN: Retry-After header
  6. Envoy v1.24.1 documentation: Adaptive Concurrency
Sign in to report a mistake