Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9524

Default window retention does not consider grace period

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.0
    • 2.8.0
    • streams
    • None

    Description

      In a windowed aggregation, if you specify a window size larger than the default window retention (1 day), Streams will implicitly set retention accordingly to accommodate windows of that size. For example, 

      .windowedBy(TimeWindows.of(Duration.ofDays(20))) 
      

      In this case, Streams will implicitly set window retention to 20 days, and no exceptions will occur.

      However, if you also include a non-zero grace period on the window, such as:

      .windowedBy(TimeWindows.of(Duration.ofDays(20)).grace(Duration.ofMinutes(5)))
      

      In this case, Streams will still implicitly set the window retention 20 days (not 20 days + 5 minutes grace), and an exception will be thrown:

      Exception in thread "main" java.lang.IllegalArgumentException: The retention period of the window store KSTREAM-KEY-SELECT-0000000002 must be no smaller than its window size plus the grace period. Got size=[1728000000], grace=[300000], retention=[1728000000]

      Ideally, Streams should include grace period when implicitly setting window retention.

      Attachments

        Issue Links

          Activity

            People

              marcolotz Marco Lotz
              mikebin Michael Bingham
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: