Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6978

Make Streams Window retention time strict

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: streams
    • Labels:
      None

      Description

      Currently, the configured retention time for windows is a lower bound. We actually keep the window around until it's time to roll a new segment. At that time, we drop all windows in the oldest segment.

      As long as a window is still in a segment, we will continue to add late-arriving records to it and also serve IQ queries from it. This is sort of nice, because it makes optimistic use of the fact that the windows live for some time after their retention expires. However, it is also a source of (apparent) non-determinism, and it's arguably better for programability if we adhere strictly to the configured constraints.

      Therefore, the new behavior will be:

      • once the retention time for a window passes, Streams will drop any later-arriving records (with a warning log and a metric)
      • likewise, IQ will first check whether the window is younger than its retention time before answering queries.

      No changes need to be made to the underlying segment management, this is purely to make the behavior more strict wrt the configuration.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vvcephei John Roesler
                Reporter:
                vvcephei John Roesler
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: