Major Description
Currently, the configured retention time for windows is a lower bound. We actually keep the window around until it's time to roll a new segment. At that time, we drop all windows in the oldest segment.
As long as a window is still in a segment, we will continue to add late-arriving records to it and also serve IQ queries from it. This is sort of nice, because it makes optimistic use of the fact that the windows live for some time after their retention expires. However, it is also a source of (apparent) non-determinism, and it's arguably better for programability if we adhere strictly to the configured constraints.
Therefore, the new behavior will be:
- once the retention time for a window passes, Streams will drop any later-arriving records (with a warning log and a metric)
- likewise, IQ will first check whether the window is younger than its retention time before answering queries.
No changes need to be made to the underlying segment management, this is purely to make the behavior more strict wrt the configuration.
Attachments
Issue Links
- links to