Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6978

Make Streams Window retention time strict

Agile BoardAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0
    • streams
    • None

    Description

      Currently, the configured retention time for windows is a lower bound. We actually keep the window around until it's time to roll a new segment. At that time, we drop all windows in the oldest segment.

      As long as a window is still in a segment, we will continue to add late-arriving records to it and also serve IQ queries from it. This is sort of nice, because it makes optimistic use of the fact that the windows live for some time after their retention expires. However, it is also a source of (apparent) non-determinism, and it's arguably better for programability if we adhere strictly to the configured constraints.

      Therefore, the new behavior will be:

      • once the retention time for a window passes, Streams will drop any later-arriving records (with a warning log and a metric)
      • likewise, IQ will first check whether the window is younger than its retention time before answering queries.

      No changes need to be made to the underlying segment management, this is purely to make the behavior more strict wrt the configuration.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vvcephei John Roesler
            vvcephei John Roesler
            Votes:
            0 Vote for this issue
            Watchers:
            4 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment