Time and Watermarks in Flink

As explained in the previous lesson, time is a crucial element that is commonly used to define boundaries on unbounded streams.

Like other stream processing systems, such as MillWheelT. Akidau et al., “MillWheel: Fault-tolerant Stream Processing at Internet Scale,” Proceedings of the VLDB Endowment, Volume 6 Issue 11, August 2013, 2013. and The Dataflow ModelT. Akidau et al., “The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-scale, Unbounded, Out-of-order Data Processing,” Proceedings of the 41st International Conference on Very Large Data Bases, Volume 8 Issue 12, August 2015, 2015., Flink supports two main notions of time: processing time and event time.

Note: Flink has a third notion of time called the ingestion time which corresponds to the time an event enters Flink. But this section will focus on event and processing time only.

Processing time

Processing time refers to the system time of the machine that is executing an operation.

Event time

Event time is the time that each event occurred on its producing device.

We can use processing time or event time with some trade-offs, which are following.

Processing time trade-offs

When a streaming program runs on processing time, all time-based operations (e.g., time windows) will use the system clock of the machines that runs the respective operation. It is the simplest notion of time and requires no coordination between the nodes of the system. It also provides good performance and reliably low latency on the produced results.

However, all this comes at the cost of consistency and non-determinism. The system clocks of different machines will differ, and the various nodes of the system will process data at different rates. As a consequence, different nodes might assign the same event to different windows depending on timing.

Event time trade-offs

When a streaming program runs on event time, all time-based operations will use the event time embedded within the stream records to track time, instead of system clocks. It brings consistency and determinism to the execution of the program since nodes will now have a common mechanism to track the progress of time and assign events to windows.

However, it requires some coordination between the various nodes, as we will see below. It also introduces some additional latency since nodes might have to wait for out of order or late events.