Backpressure and load shedding
Two ways to survive a sender that outruns a receiver: backpressure makes the sender slow down, and load shedding rejects excess requests early and cheaply so the work you do accept still finishes in time.
What it is
Backpressure is what happens when data piles up behind a buffer because the receiving side is slower than the sending side, and it is also the name for the fix: a signal from the receiver that the sender honors by pausing or slowing down. Unix pipes and TCP sockets do this and call it flow control. In Node.js streams the signal is the return value of a writable stream's write() call, which tells the source whether it may keep sending.
Load shedding is the other response: as a server approaches overload it drops some share of the incoming traffic on purpose, so that it still completes the most useful work possible instead of running out of memory, failing its health checks or answering so slowly that the answers are useless. A simple form is a bounded queue in front of the worker threads that rejects new requests when it is full; another is returning an HTTP 503 once more than a set number of requests are in flight.
Why a server needs either one: overload amplifies itself. Throughput is the total number of requests arriving; goodput is the part answered without error and fast enough for the client to use. Once a server's median latency reaches the client timeout, half of its responses arrive after the client has given up, so a latency problem becomes an availability problem. Each timed-out request wastes the work already done on it, and clients often retry, which multiplies the offered load. In a deep call chain where each layer retries, the multiplication compounds at each hop. Shedding early keeps goodput roughly flat while offered load keeps climbing, until the cost of rejecting itself becomes the bottleneck.
| Mechanism | How it signals | Where you meet it |
|---|---|---|
Flow control | The receiver tells the sender how much it can take; the sender pauses | TCP, Unix pipes, stream libraries |
Bounded queue | The queue refuses new work when full | The request queue in front of a worker pool |
Fast rejection | A | Overloaded web servers and proxies |
Latency-driven limit | Latency samples raise or lower the number of in-flight requests allowed | Envoy 1.24.1's adaptive concurrency filter |
Deadline propagation | Each hop passes on the time remaining; work stops when it runs out | RPC frameworks, request queues |
Graceful degradation | Cheaper, lower-quality answers instead of refusals | Search over a cache subset, simpler ranking |
Queues are where overload hides. A queued request costs memory and adds waiting time. If the queue holds ten times as many requests as there are threads and each request takes 100 ms of worker time, a request that enters a full queue takes about 1.1 seconds, nearly all of it waiting. A queue that is small relative to the thread pool, about half its size or less, makes the server say no early when it can't keep up. Bursty traffic is the exception where a deeper queue, sized from the threads in use, the per-request time and the burst shape, can be worth it.
Size is not the whole story: bound the time a request may sit in a queue too, and throw it out when that bound passes, because by the time a long-queued request reaches a worker the client may have timed out. Record how long each dequeued request waited. In-memory queues also turn up in places you didn't put them, such as the executor inside a web framework and the operating system's socket buffers, so assume there are queues you haven't found and load-test past the breaking point to see how they behave. A spillover setting that fails fast instead of queuing excess is the safer default; Amazon's Application Load Balancer rejects excess traffic where the older Classic Load Balancer kept a surge queue.
What to drop first. The one request to protect is the load balancer's health check: miss it and the balancer stops sending traffic to that server, shrinking the fleet in the middle of a brownout. Beyond that, prioritize by who is asking and by sunk cost: a human's request over a crawler's, the call that finishes a multi-step job over the call that starts a new one, and page N of a paginated read over page one, since dropping late means throwing away everything already spent. Make these decisions as early in the stack as possible, in the service that first receives the request, rather than guessing at priorities deep inside.
Watch the clock. A request whose client has already given up is wasted work, so a server that notices the deadline has passed partway through can stop. The deadline is set once, high in the stack, and each hop passes on what remains: a 30-second deadline that spends 7 seconds in server A reaches server B with 23 seconds left. Measure durations with a monotonic clock, since wall clocks jump when they resync; and remember that the stopwatch may already be late, because a request can sit in a TCP buffer before the server reads it. Related to this, a backend that is known to be down should fail immediately rather than holding a worker until the deadline, or a small fraction of hanging requests can tie up most of the worker threads.
Adaptive limits replace a hand-tuned number with a control loop. Envoy 1.24.1's adaptive concurrency filter periodically measures an ideal round-trip time (minRTT) by allowing a very small number of outstanding requests to the upstream, then compares sampled latencies against it. It computes a gradient, (minRTT + B) / sampleRTT where B is a buffer that tolerates normal variance, and sets the new limit to gradient times the old limit plus a headroom pinned to the square root of the limit. As sampled latency rises the gradient shrinks, which pulls the limit down once the drop outweighs the headroom; while latency stays near minRTT the headroom keeps nudging the limit up. The documentation warns that 503 responses rise during the minRTT window, because the limit drops to 3 by default, and recommends retries to another host.
When to use it
Reach for backpressure when the sender can hold its data and wait: a file being compressed to disk, a worker pulling from a log at its own pace, a stage in a pipeline. The Node.js guide shows what the signal is worth: compressing a file while respecting the write() return value peaked at about 88 MB of memory, and ignoring it peaked at about 1.52 GB for the same job. That is the general shape of the win: the sender's pace is capped by the receiver's, so buffers stay bounded.
Reach for load shedding when the sender can't wait or the request loses its value as it ages. A user whose search has been queued for ten seconds has probably refreshed the page and sent another one, so answering the first is pointless; serving the newest requests first (last in, first out) or a controlled-delay queue discipline tends to shed the long-waiting requests that are least likely to still matter, and works best when deadlines travel with the request. Return 503 for the rejected ones, since that code is meant for temporary conditions and pairs with a Retry-After header carrying either a date or a number of seconds to wait. Don't let caches store the 503, or clients may keep seeing the error page after recovery.
Prefer graceful degradation over outright refusal when a cheaper answer still helps the user: searching a cached subset instead of the full index, or a faster, less accurate ranking. Shared services serving many clients are where selective shedding by client or by request importance pays off most.
When not to
Don't make shedding the first line of defense against load you can predict. Capacity and automatic scaling come first; shedding is for the surge, the lost capacity after a bad deploy, or the day clients switch from cheap cached reads to expensive misses. Degraded modes should trip rarely, and a mode that rarely runs is a mode you have little experience operating, so keep it simple, exercise it on purpose by running a few servers near overload, and build a quick way to turn it off, since complex degradation logic can itself push a server into degraded mode or into a feedback cycle at the wrong moment.
Don't drop a request when dropping is dearer than serving it. Dropping has a cost, and a stray log line or socket setting can make it much larger than expected; load-test far past the plateau to find out. In the rare case where rejecting is the more expensive path, slowing rejected requests down to at least the latency of a success, while they hold no application thread, beats a fast refusal.
Don't treat a per-client rate limit or a proxy's connection cap as the whole answer to overload. A per-client rate limit doesn't look at the service's health, so it may not stop a failure already under way, and a simple one tends to leave capacity idle. A max_conns style cap on a proxy does shed load, but it is hard to prioritize by request importance and the raw in-flight count can misjudge whether the service is in trouble, which is why it belongs as a last resort rather than the default guard. Early rejection at a proxy or firewall is the cheapest place to drop, but you lose visibility into what you dropped, so protect in layers and log enough at each one.
And don't count on backpressure across a boundary where the producer can't pause: a person clicking a button, a partner's webhook, a device that emits readings on a timer. The signal has nowhere useful to land, so the buffer just moves one hop upstream. Our advice: at that boundary, accept into a bounded, durable queue or shed; use backpressure between the stages you control behind it.
Compared
| Option | What it does | Pick it when | A simpler option is right when |
|---|---|---|---|
Backpressure (flow control) | Receiver paces the sender; buffers stay bounded | Sender can pause and hold its data: pipelines, log consumers, file transfers | The sender can't pause, so the signal has no effect; shed or buffer durably instead |
Bounded queue with fast reject | Refuses new work when the queue or in-flight count hits a fixed limit | Steady traffic and a known per-request cost; you want a cheap, predictable guard | Capacity or the cost per request varies, so a fixed number is too low at some times and too high at others |
Latency-driven in-flight limit | Moves the in-flight cap up and down from latency samples | Upstream capacity shifts, per-request cost varies, and you can route all traffic through the filter | Some traffic bypasses the control point; a fixed limit is easier to reason about and debug |
Deadline-based dropping | Discards requests whose client has already given up | Deep call chains where late answers are worthless | Single-hop services with short, uniform latency, where a bounded queue already catches it |
Graceful degradation | Serves a cheaper answer instead of refusing | A partial or approximate answer still helps the user | There is no useful cheaper answer (a payment, a write), so a clear refusal is more honest |
Rate limiting per client | Caps each caller's request rate | Fairness between tenants and abuse control | Overload comes from legitimate load the limit can't see, so it neither stops it nor uses idle capacity |
In a picture
Step 1 of 3
The unprotected path
Requests flow through a load balancer into a request queue, onto worker threads, and to the database. With no bound on the queue, a surge makes it grow, waiting time passes the client timeout, and retries add to the pile.