Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Whenever we aggregate data by event time, we want to consider data is late and out-of-order in terms of its event time. Since we keep aggregate keyed by the time as state, the state will grow unbounded if we keep around all old aggregates in an attempt consider arbitrarily late data. Since the state is a store in-memory, we have to prevent building up of this unbounded state. Hence, we need a watermarking mechanism by which we will mark data that is older beyond a threshold as “too late”, and stop updating the aggregates with them. This would allow us to remove old aggregates that are never going to be updated, thus bounding the size of the state.
Here is the design doc - https://docs.google.com/document/d/1z-Pazs5v4rA31azvmYhu4I5xwqaNQl6ZLIS03xhkfCQ/edit?usp=sharing
