Skip to content

Stream processing and windows

A window cuts an endless stream into finite slices so a count or sum can finish; event time and watermarks decide when a slice is complete, and allowed lateness decides what happens to data that arrives late.

On this page

What it is

Stream processing computes over events as they arrive, instead of over a data set that is already complete. Some operations don't mind that the stream has no end: a map or a filter looks at one record and moves on, and a rolling count per key can keep one running total forever without a window. The moment the question has a period in it (orders per hour, visitors per session), the processor has to cut the stream into finite groups. That cut is a window. A stateful operator keeps a running result per key and per window in a state store, and emits it when the window fires, usually once the window is judged complete; a window can fire again if late data is accepted.

WindowHow it is boundedAn event lands inTypical question

Tumbling

A fixed size; windows sit side by side without overlap

Exactly one window

How many orders per hour?

Sliding (Kafka Streams says hopping)

A size and a slide; when the slide is shorter than the size, windows overlap

Several windows, one copy of the event per window

Average over the last 5 minutes, refreshed each minute

Session

A gap of inactivity; an event inside the gap extends the session, and windows that come within the gap of each other merge

The session for its key

How long did this visit last?

Global

No time bound; one window per key that stays open, with a trigger you supply deciding when it fires and whether its contents are purged

The single window for its key

Fire after N events, or on a custom condition

A naming trap: in Kafka Streams, hopping is the size-plus-advance window, a tumbling window is a hopping window whose advance equals its size, and sliding means something else, a window defined by the largest time difference allowed between two events. Other frameworks use sliding for the size-plus-slide window. In a round, say the shape (size, slide, gap) rather than relying on the name.

Two clocks can drive a window. Processing time is the wall clock of the machine doing the work. It is simple and adds no waiting, but the result depends on how fast records reach the system and move between operators, so a replay of the same input can give a different answer. Event time is the timestamp each record carries from the device that produced it. Progress depends on the data rather than a wall clock, so an hourly window holds the records whose timestamps fall in that hour, whatever order they arrived in. A third clock, ingestion time, is the moment the processor first saw the record, a middle ground between the two.

Event time raises a question: how does the operator know the hour is over when a straggler might still show up? The answer is a watermark, a marker that travels inside the stream and carries a timestamp t. It states that event time has reached t, so elements with timestamps at or before t should be done arriving. When a watermark reaches a window operator, the operator moves its event-time clock forward and closes any window whose end is now behind it. In Flink 2.1, watermarks are produced at or right after the sources, with each parallel source usually emitting its own, and a common strategy assumes out-of-order arrival is bounded by a maximum delay, so the watermark trails the newest timestamp seen by that amount.

A watermark is a bet, and some elements lose it: they arrive after the watermark has passed their timestamp. These are late. You choose what happens to them. Flink's window operators have an allowed lateness that defaults to zero, so an element that arrives once the watermark is past the end of its window is dropped. Raise it and the window's state is kept for that extra time, a late element is added, and with the default event-time trigger the window fires again with an updated result. Elements later than that can be routed to a side output instead of being lost. Kafka Streams calls the same idea a grace period: a window keeps accepting records that belong to it for that long after stream time has passed its end. The window and its state are removed once time passes the end plus the allowed lateness: a tumbling window for 12:00 to 12:05 with one minute of allowed lateness is created when its first element arrives and removed when the watermark passes 12:06.

When to use it

Reach for a window whenever a running aggregate over a stream would otherwise grow without bound: counts, sums, averages, top-k per period, and joins between two streams. Kafka Streams makes a stream-to-stream join windowed for that reason: without a window, the buffer of records waiting for a match would keep growing.

Pick tumbling when each event should count once and the report has natural periods: hourly revenue, errors per minute. Pick sliding when the question is "the last N minutes" asked more often than N, as in a moving average or an alert on a rolling error rate. Pick session when the boundary is the user's own pause: a visit, a game session, a support chat. Use event time when the result has to be reproducible; the Flink tutorial gives the highest price in the first hour of trading as a case where event time is the right clock.

Set the wait from the data. Measure how late events typically arrive from each source (mobile clients that were offline, a partner feed that batches), then set the watermark delay and the allowed lateness from that, and route the rest to a side output you reconcile later. Holding watermarks back far enough to catch the slowest straggler delays the results of all windows, and some elements are delayed by so much that no finite wait catches them.

When not to

If the question is answered once a day over data that has finished arriving, a batch job over the stored events is simpler: no watermarks, no late data, and a rerun gives the same answer. Windows earn their complexity when someone needs the result while the stream is still flowing.

Processing time is the right clock when you want to know what the system is handling right now and a replay doesn't have to match: throughput dashboards, operational alerting, measuring the current rate. It skips the waiting that event time adds for out-of-order events, and nothing is late because the clock is the operator's own wall clock rather than a timestamp in the data.

Avoid a sliding window whose slide is far smaller than its size over heavy traffic: an element is assigned to each window it overlaps, so a long window with a tiny slide multiplies both state and output. Aggregate per small tumbling window and combine the recent results downstream instead.

A global window with a custom trigger is rarely the answer. If the need is "after 100 events", say so and justify it. If the need is time-based, a tumbling or session window also cleans up its own state; a global window has no natural end, so nothing expires on its own and you write the trigger and the cleanup yourself.

Compared

WindowBoundariesState heldRight choice whenWeaker when

Tumbling

Fixed size, no overlap

One element, or one accumulator, per window it belongs to

Periodic reports where each event counts once

You need "the last N minutes" more often than once per N

Sliding (hopping)

Fixed size plus a slide; windows overlap

One entry per overlapping window, as an element copy or an accumulator

Moving averages and rolling alerts

The slide is tiny next to the size; state and output multiply

Session

A gap of inactivity; set by the data, and windows merge

Per active key until the session closes and any allowed lateness has passed

Behavior with natural pauses, such as visits

A key rarely pauses, so its session keeps growing

Global

None by time; a trigger you write

Kept until a trigger purges it; a plain fire keeps the contents, and an incremental accumulator keeps it small

Count-based or custom firing

Anything time-based: it has no natural end, so nothing expires on its own

In a picture

Stamped eventsKey and windowWindow state

Step 1 of 3

Key the stream and assign windows

Each event carries an event-time stamp. The operator groups events by key and assigns each one to a window; the running result lives in per-key, per-window state.

How to say it in an interview

The common mistake

Quiz

An event stamped 10:04 arrives after the watermark has passed 10:05 and the 10:00 to 10:05 window has fired. Allowed lateness is zero. What happens to it under the defaults?
You need the average latency over the last 5 minutes, refreshed twice a minute. Which window fits?
What does a watermark carrying timestamp t claim?
Which clock gives the same windowed result when you replay the same input?

Questions that use this

Related

Sources

  1. Apache Flink 2.1 documentation: Timely stream processing
  2. Apache Flink documentation (stable release): Windows
  3. Apache Flink documentation (stable release): Windows in the DataStream API v2
  4. Apache Flink documentation (stable release): Streaming analytics
  5. Apache Kafka 4.1 documentation: Kafka Streams DSL
  6. Confluent Developer: Windowing in Kafka Streams
Sign in to report a mistake