Skip to content

Streaming LLM tokens

Sending a model's answer as small chunks over one open HTTP response while it is still being generated, so the reader sees the first words long before the last. The design questions are the transport, two latency numbers and disconnects.

On this page

What it is

A decoder-style language model produces its output one token at a time. It first processes the prompt (prefill), which builds the key-value cache the attention layers need, then enters a loop (decode) where each step predicts the next token from everything so far. Streaming returns partial output as chunks of tokens while that loop is still running; without it, the client receives the complete answer in one body at the end. A token is roughly three quarters of an English word for many popular models, so a 400-word reply is several hundred decode steps.

Two numbers matter most to the person reading a stream. Time to first token (TTFT) covers queueing, prefill and network time, and it grows with prompt length because the whole input has to be processed before the first token exists. Inter-token latency (ITL, also called time per output token) is the average gap between consecutive tokens; GenAI-Perf leaves the first token out of that average so the number reflects just the decode phase. End-to-end latency is TTFT plus the generation time, so streaming mostly changes when the reader sees the first part rather than when the last token lands; throughput figures such as tokens per second and requests per second describe the server's side of the same picture. During decode the cache and the per-token attention cost grow with the sequence length so far, so a steady ITL is a sign the server manages memory well.

On the wire, the transport this page uses is server-sent events (SSE), which the Claude Messages API streams over. The server answers with the MIME type text/event-stream and writes each event as a block of text ended by two newlines, using the fields event, data, id and retry. In a browser, EventSource opens a persistent connection, fires an event named after the event field (or a generic message event when it's absent), and stays open until close() is called. The channel is one-way, server to client. retry tells the browser how many milliseconds to wait before reconnecting after a lost connection, and id sets the client's last event ID. Without HTTP/2, a browser allows about six open connections per domain across all tabs; over HTTP/2 the stream limit is negotiated and defaults to 100.

http
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
X-Accel-Buffering: no

id: 1
event: start
data: {"response":"resp_01"}

id: 2
event: delta
data: {"text":"The cache"}

id: 3
event: delta
data: {"text":" sits in front of"}

id: 4
event: done
data: {"stop":"end_turn"}

An event protocol sits on top of SSE. In the Claude Messages API, setting "stream": true produces a message_start, then one or more content blocks, each with a start event, any number of content_block_delta events and a stop event, each carrying an index into the final content array; then message_delta events for top-level changes and a closing message_stop. ping events may appear at any point, and an error can arrive as an event: an overloaded_error in the stream corresponds to an HTTP 529 on a non-streaming call. New event types may be added over time, so client code should skip types it doesn't recognize. Tool-call arguments stream as fragments of a JSON string, while the final input is an object, so a client accumulates the fragments and parses at the block's stop event (or uses a partial-JSON parser). Token counts in message_delta are cumulative, not per event.

When to use it

Stream when a person is reading while the model writes: chat, coding assistants, drafting tools. The gain is at the start: the reader begins on the opening sentence while later chunks arrive in the background. It also removes a timeout problem: Anthropic's SDKs require streaming for requests with large max_tokens values, because a single HTTP request waiting for the full result can exceed the timeout.

Our advice for the design around the stream. Persist as you send: append each chunk, with a sequence number, to a store keyed by the response ID; put that number in the SSE id field, and have the reconnect handler accept the last ID the client saw and replay the chunks after it, so a dropped Wi-Fi link doesn't restart the answer. Cancel on disconnect: the MDN server example breaks its loop when it finds the client has gone; do the same and cancel the upstream generation, which frees GPU time for other users. Turn off intermediary buffering: the same example sends X-Accel-Buffering: no, because a proxy that buffers the body turns a stream into one late block. Coalesce: the inference server emits chunks of tokens; we'd send a few tokens per event rather than one, since one event per token adds framing overhead with little visible benefit.

When not to

Streaming helps less when the consumer is a program that needs the whole result: a service that parses one JSON object, a classifier returning a label, a pipeline that writes the finished record. Tool-call arguments arrive as string fragments that become an object at the block's end, so unless the consumer uses a partial-JSON parser, it waits for the block anyway. Short outputs are a second case: when the answer is a handful of tokens, end-to-end latency is close to TTFT, and streaming adds event handling for no visible gain. Plain request and response is the right tool there.

Consider something other than SSE when the client must speak often mid-stream, as in voice assistants where the user interrupts, or shared editing sessions: SSE carries data in one direction, so a cancel or a steer has to travel as a separate request, and a WebSocket fits traffic that flows both ways. Be wary of opening many streams per page on HTTP/1.1, where the browser's connection limit is about six per domain. If output must be reviewed before anyone sees it, streaming works against you: you'd buffer by sentence or paragraph and check each piece, which gives up part of the latency win. And for work that runs for minutes, a background job the client polls or re-attaches to is usually sturdier than a connection held open the whole time.

Compared

OptionHow tokens reach the clientPick it whenIt's the weaker choice when

Wait for the whole response

One JSON body after the last token

A program consumes the result, or the output is short

A person is waiting through a long answer

Server-sent events

One HTTP response, text/event-stream, an event per chunk, browser reconnects automatically

A person reads as the model writes, and traffic is server to client

The client must send mid-stream (interrupting, voice), or many streams per page over HTTP/1.1

WebSocket

Messages in both directions on one connection

The user interrupts or steers generation while it runs

You need one-way text and would rather keep plain HTTP requests and the proxies you already run

Background job, then poll or re-attach

Server generates and stores the answer; client fetches or subscribes later

Runs last minutes, or clients drop often

A chat reply measured in seconds, where polling adds delay

In a picture

POST promptstream: trueprefilldecode loopchunksSSE eventsappendBrowserChat APIInference serverFirst tokenNext tokensResponse log
The API holds one response open: the model prefills the prompt, then decodes tokens that the API forwards as events and appends to a log for resume.

How to say it in an interview

The common mistake

Quiz

Users start pasting long documents into the prompt, while the answers stay about the same length. Which number moves most?
A user closes the tab halfway through a 2,000-token answer. What should the server do?
Tool-call arguments are arriving as fragments of a JSON string. When can a standard JSON parser read them?
Which SSE field gives a client the position to resume from after a dropped connection?

Related

Sources

  1. Claude Platform Docs: Streaming messages
  2. MDN: Using server-sent events
  3. MDN: EventSource
  4. NVIDIA Technical Blog: LLM Inference Benchmarking: Fundamental Concepts
Sign in to report a mistake