Skip to content

GPU batching and scheduling

How an inference server groups requests into one GPU pass and decides, step by step, which requests run next, so one model copy serves many users without short requests waiting behind long ones.

On this page

What it is

A GPU runs one forward pass of a model over a batch of inputs. For a large model, much of the time in that pass goes to moving the weights from GPU memory into the compute units, and that cost is paid once per pass however many inputs ride along. Batching several requests into one pass spreads that memory traffic across them. The authors of Hugging Face's Text Generation Inference (TGI) describe LLM generation as memory-bound, with compute to spare relative to the memory it needs, and batching is what turns that spare compute into throughput.

The simplest scheduler works on whole requests. In NVIDIA Triton Inference Server, enabling dynamic_batching in a model's configuration lets the server combine separate requests into one batch, up to the model's maximum batch size. With the default settings it batches whatever is already waiting and runs at once; max_queue_delay_microseconds lets the scheduler hold the batch for a bounded time so more requests can join. The user guide's worked example turns five requests that take 5X ms one after another into 3X ms batched, and it notes that real batches don't parallelize perfectly, so a bigger batch takes somewhat longer per pass. The guide aims this batcher at stateless models such as object detectors; stateful ones go through a separate sequence batcher. Triton can also run several instances of a model, on one GPU or across GPUs, through instance_group, so batches execute concurrently.

Text generation is different. The model runs once per output token, so a request is many iterations long and requests finish at different times. With request-level scheduling, a sequence that has finished stays in the batch, holding its slot, until the longest one is done, and a request that arrives mid-batch waits for the whole batch to end. Orca (OSDI 2022) names this problem and proposes iteration-level scheduling: the scheduler invokes the engine for a single iteration at a time, so after each token step it can drop finished requests and admit new ones. Because sequences in the batch are at different lengths, it pairs this with selective batching, which batches a chosen subset of the model's operations across requests and handles the others request by request. On a GPT-3 175B model the authors report 36.9 times the throughput of FasterTransformer at the same latency.

Serving engines call this continuous batching. In TGI's router, a new request is merged into the running batch as soon as the current forward pass finishes, and a sequence that emits its end token or reaches its limit is dropped and its memory freed. The first pass over a prompt, called prefill, computes attention for the whole prompt and costs far more than each later pass, called decode, which handles one new token. A new request's prefill has to run before it can join the decode group, and while it runs, the sequences already in flight produce no tokens. vLLM 0.8.2 offers chunked prefill for this: a long prompt is split into chunks that are batched with decode steps under a token budget, max_num_batched_tokens, which defaults to 2048 in that version. A smaller budget gives better inter-token latency because fewer prefill tokens interrupt decodes.

What caps the batch is GPU memory, not a request count. Each sequence keeps a key-value (KV) cache of attention state that grows by one token per step and is freed when the sequence ends. The PagedAttention paper reports that this per-request memory is large, that it grows and shrinks over a request's life, and that fragmentation and duplication in earlier systems wasted enough of it to limit batch size: those systems set aside one contiguous chunk per request sized to its maximum length, 2048 tokens for example, so a request that stopped early left most of that chunk unused. PagedAttention allocates the cache in fixed-size blocks, in the manner of virtual-memory pages, with near-zero waste, and the authors measured 2 to 4 times the throughput of earlier serving systems at the same latency.

When traffic exceeds what fits, the scheduler has to pick. vLLM 0.8.2 preempts requests to free KV cache for others and recomputes the preempted ones once space is available again. The preempted request pays in latency and the recomputation spends GPU time, so admission control upstream is cheaper than preemption inside the engine.

When to use it

Use dynamic batching for stateless models under concurrent traffic: image classifiers, embedding models, rerankers, anything where one request is one forward pass. Our advice: set the queue delay to a small fraction of the latency budget, since a request that arrives first can wait the whole delay before its batch launches, and start from the default of no delay if traffic is already dense enough to fill batches on its own.

Use continuous batching for autoregressive generation served online to many users: chat, code completion, summarization. The TGI authors frame the result as a middle ground, with throughput far above one request at a time and latency far better than waiting to fill a static batch; they consider doubling latency to serve ten times the users on the same hardware a fair trade. Add chunked prefill when long prompts share the server with interactive decodes, so one user's long document holds up everyone else's next token for less time. Pair the scheduler with admission control, such as backpressure and load shedding, so that overload turns into queueing or rejection at the edge rather than preemption inside the engine.

When not to

Skip the batching delay when traffic is light or a single request's latency is the goal. Holding a request for a queue delay helps if another request is likely to arrive in that window; if none does, the delay is pure waiting. The dynamic batcher's default is to batch whatever is present and run at once, and the TGI authors describe running with no batching as the setting with the best latency and a throughput near one request at a time. Our advice: for a single-user tool or an on-device model, serve one request at a time and skip the batcher.

Skip continuous batching for offline work. For a nightly job that embeds or summarizes a corpus, no one is waiting on a token, so static batches at a size the hardware handles well can approach its throughput ceiling with far less machinery; measure a few sizes, since a larger batch is not automatically faster per item. Stateful models, where requests from one session must land on the same instance in order, don't fit a plain dynamic batcher either; a separate sequence batcher handles those.

Don't assume bigger is better. A larger batch takes longer per pass because not all of its work parallelizes, so per-token latency rises while the throughput gained per added request shrinks. Continuous batching also pays off most when the model has compute headroom relative to its memory traffic; a small, compute-bound model gains less from it. Tune the max batch size against the latency budget and measure, rather than setting it to the largest value that fits.

Compared

ApproachHow a batch formsWhat it costsWhen it is the right choice

No batching

Each request gets its own forward pass

No queue wait when traffic is light; throughput of about one request at a time, and under load requests queue behind each other

A single user, an on-device model, or a strict latency target with light traffic

Static batching

The client, or an offline job, sends a fixed-size batch

Can approach the hardware's throughput ceiling; requests wait until the batch is full and all finish together

Offline and bulk work where no one is waiting on a single result

Dynamic batching

The server merges requests that arrive within a bounded delay, up to a max size

Requests may wait up to the queue delay for others to join; a finished request still waits for the batch

Stateless models with concurrent online traffic: classifiers, embeddings, rerankers

Continuous batching

The scheduler rebuilds the batch at each token step; finished sequences leave, new ones join

Per-token latency rises with batch size; a new prompt's prefill interrupts running decodes unless chunked

Autoregressive generation served online to many concurrent users

In a picture

admititeratetokenyesnoNew requestsRequest poolSchedulerGPU forward passFinished?Return to client
Iteration-level scheduling: after each token step the scheduler drops finished sequences and admits waiting ones whose KV cache fits.

How to say it in an interview

The common mistake

Quiz

Why does batching several sequences into one decode step raise an LLM server's throughput so much?
Users of a chat service notice that short answers arrive late whenever someone else asks for a long one. The server batches whole requests. What fixes it?
In an LLM serving engine, what usually caps how many sequences can be in flight?
An embedding model gets a few requests per second with a tight latency target. What does adding a 10 ms dynamic-batching queue delay most likely do?

Related

Sources

  1. NVIDIA Triton Inference Server user guide: Dynamic batching and concurrent model execution
  2. Hugging Face Text Generation Inference: router README (continuous batching)
  3. USENIX OSDI 2022: Orca, a distributed serving system for Transformer-based generative models (abstract)
  4. arXiv: Efficient Memory Management for Large Language Model Serving with PagedAttention (abstract)
  5. ar5iv: Efficient Memory Management for Large Language Model Serving with PagedAttention (full text)
  6. vLLM 0.8.2 documentation: Optimization and Tuning
Sign in to report a mistake