Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3658

Incorrect validation check on maintenance period with join window size

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0.0
    • Fix Version/s: 0.10.0.0
    • Component/s: streams
    • Labels:

      Description

      As Henry Cai found out, the current validation check of KStreamJoinWindow requires the retention period to be at least twice than the join window size. This check was originally for making the segment interval to be larger than the join window size. But for windowed stream-stream join this is not necessary.

      More specifically, for example with a window size 6, and retention period 12, and num. segment 5, the segment size will be set to 3. This means after time 12, the first segment of [0, 3) will be dropped, then at time 13, a late record with timestamp (1) will not be accepted to the window store, and will not participate in the joining as well.

      The proposed change is to only require retention period to be > window size, not window size * 2.

      cc Yasuhiro Matsuda

        Attachments

          Activity

            People

            • Assignee:
              guozhang Guozhang Wang
              Reporter:
              guozhang Guozhang Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: