Stream processing and windows
A window cuts an endless stream into finite slices so a count or sum can finish; event time and watermarks decide when a slice is complete, and allowed lateness decides what happens to data that arrives late.
What it is
Stream processing computes over events as they arrive, instead of over a data set that is already complete. Some operations don't mind that the stream has no end: a map or a filter looks at one record and moves on, and a rolling count per key can keep one running total forever without a window. The moment the question has a period in it (orders per hour, visitors per session), the processor has to cut the stream into finite groups. That cut is a window. A stateful operator keeps a running result per key and per window in a state store, and emits it when the window fires, usually once the window is judged complete; a window can fire again if late data is accepted.
| Window | How it is bounded | An event lands in | Typical question |
|---|---|---|---|
Tumbling | A fixed size; windows sit side by side without overlap | Exactly one window | How many orders per hour? |
Sliding (Kafka Streams says hopping) | A size and a slide; when the slide is shorter than the size, windows overlap | Several windows, one copy of the event per window | Average over the last 5 minutes, refreshed each minute |
Session | A gap of inactivity; an event inside the gap extends the session, and windows that come within the gap of each other merge | The session for its key | How long did this visit last? |
Global | No time bound; one window per key that stays open, with a trigger you supply deciding when it fires and whether its contents are purged | The single window for its key | Fire after N events, or on a custom condition |
A naming trap: in Kafka Streams, hopping is the size-plus-advance window, a tumbling window is a hopping window whose advance equals its size, and sliding means something else, a window defined by the largest time difference allowed between two events. Other frameworks use sliding for the size-plus-slide window. In a round, say the shape (size, slide, gap) rather than relying on the name.
Two clocks can drive a window. Processing time is the wall clock of the machine doing the work. It is simple and adds no waiting, but the result depends on how fast records reach the system and move between operators, so a replay of the same input can give a different answer. Event time is the timestamp each record carries from the device that produced it. Progress depends on the data rather than a wall clock, so an hourly window holds the records whose timestamps fall in that hour, whatever order they arrived in. A third clock, ingestion time, is the moment the processor first saw the record, a middle ground between the two.
Event time raises a question: how does the operator know the hour is over when a straggler might still show up? The answer is a watermark, a marker that travels inside the stream and carries a timestamp t. It states that event time has reached t, so elements with timestamps at or before t should be done arriving. When a watermark reaches a window operator, the operator moves its event-time clock forward and closes any window whose end is now behind it. In Flink 2.1, watermarks are produced at or right after the sources, with each parallel source usually emitting its own, and a common strategy assumes out-of-order arrival is bounded by a maximum delay, so the watermark trails the newest timestamp seen by that amount.
A watermark is a bet, and some elements lose it: they arrive after the watermark has passed their timestamp. These are late. You choose what happens to them. Flink's window operators have an allowed lateness that defaults to zero, so an element that arrives once the watermark is past the end of its window is dropped. Raise it and the window's state is kept for that extra time, a late element is added, and with the default event-time trigger the window fires again with an updated result. Elements later than that can be routed to a side output instead of being lost. Kafka Streams calls the same idea a grace period: a window keeps accepting records that belong to it for that long after stream time has passed its end. The window and its state are removed once time passes the end plus the allowed lateness: a tumbling window for 12:00 to 12:05 with one minute of allowed lateness is created when its first element arrives and removed when the watermark passes 12:06.
When to use it
Reach for a window whenever a running aggregate over a stream would otherwise grow without bound: counts, sums, averages, top-k per period, and joins between two streams. Kafka Streams makes a stream-to-stream join windowed for that reason: without a window, the buffer of records waiting for a match would keep growing.
Pick tumbling when each event should count once and the report has natural periods: hourly revenue, errors per minute. Pick sliding when the question is "the last N minutes" asked more often than N, as in a moving average or an alert on a rolling error rate. Pick session when the boundary is the user's own pause: a visit, a game session, a support chat. Use event time when the result has to be reproducible; the Flink tutorial gives the highest price in the first hour of trading as a case where event time is the right clock.
Set the wait from the data. Measure how late events typically arrive from each source (mobile clients that were offline, a partner feed that batches), then set the watermark delay and the allowed lateness from that, and route the rest to a side output you reconcile later. Holding watermarks back far enough to catch the slowest straggler delays the results of all windows, and some elements are delayed by so much that no finite wait catches them.
When not to
If the question is answered once a day over data that has finished arriving, a batch job over the stored events is simpler: no watermarks, no late data, and a rerun gives the same answer. Windows earn their complexity when someone needs the result while the stream is still flowing.
Processing time is the right clock when you want to know what the system is handling right now and a replay doesn't have to match: throughput dashboards, operational alerting, measuring the current rate. It skips the waiting that event time adds for out-of-order events, and nothing is late because the clock is the operator's own wall clock rather than a timestamp in the data.
Avoid a sliding window whose slide is far smaller than its size over heavy traffic: an element is assigned to each window it overlaps, so a long window with a tiny slide multiplies both state and output. Aggregate per small tumbling window and combine the recent results downstream instead.
A global window with a custom trigger is rarely the answer. If the need is "after 100 events", say so and justify it. If the need is time-based, a tumbling or session window also cleans up its own state; a global window has no natural end, so nothing expires on its own and you write the trigger and the cleanup yourself.
Compared
| Window | Boundaries | State held | Right choice when | Weaker when |
|---|---|---|---|---|
Tumbling | Fixed size, no overlap | One element, or one accumulator, per window it belongs to | Periodic reports where each event counts once | You need "the last N minutes" more often than once per N |
Sliding (hopping) | Fixed size plus a slide; windows overlap | One entry per overlapping window, as an element copy or an accumulator | Moving averages and rolling alerts | The slide is tiny next to the size; state and output multiply |
Session | A gap of inactivity; set by the data, and windows merge | Per active key until the session closes and any allowed lateness has passed | Behavior with natural pauses, such as visits | A key rarely pauses, so its session keeps growing |
Global | None by time; a trigger you write | Kept until a trigger purges it; a plain fire keeps the contents, and an incremental accumulator keeps it small | Count-based or custom firing | Anything time-based: it has no natural end, so nothing expires on its own |
In a picture
Step 1 of 3
Key the stream and assign windows
Each event carries an event-time stamp. The operator groups events by key and assigns each one to a window; the running result lives in per-key, per-window state.
How to say it in an interview
The common mistake
Quiz
Questions that use this
Related
Sources
- Apache Flink 2.1 documentation: Timely stream processing
- Apache Flink documentation (stable release): Windows
- Apache Flink documentation (stable release): Windows in the DataStream API v2
- Apache Flink documentation (stable release): Streaming analytics
- Apache Kafka 4.1 documentation: Kafka Streams DSL
- Confluent Developer: Windowing in Kafka Streams