Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.10.0.0
Description
As hcai@pinterest.com found out, the current validation check of KStreamJoinWindow requires the retention period to be at least twice than the join window size. This check was originally for making the segment interval to be larger than the join window size. But for windowed stream-stream join this is not necessary.
More specifically, for example with a window size 6, and retention period 12, and num. segment 5, the segment size will be set to 3. This means after time 12, the first segment of [0, 3) will be dropped, then at time 13, a late record with timestamp (1) will not be accepted to the window store, and will not participate in the joining as well.
The proposed change is to only require retention period to be > window size, not window size * 2.
cc ymatsuda