[BEAM-3776] StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.2.0, 2.3.0, 2.4.0
• Fix Version/s: 2.5.0
• Component/s: runner-core
• Assignee: Sam Whittle
• Reporter: Sam Whittle
• Created: 03/Mar/18 00:34
• Updated: 16/May/20 13:28
• Resolved: 18/May/18 17:45

Description

WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take to not add holds that would be before the input watermark.

However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the window depends only on window, sets a hold for the end of the window regardless of the input watermark.

Thus if you have a WindowingStrategy such as:

WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
.withMode(AccumulationMode.DISCARDING_FIRED_PANES)
.withTrigger(
Repeatedly.forever(
AfterWatermark.pastEndOfWindow()
.withLateFirings(AfterPane.elementCountAtLeast(10))))
.withAllowedLateness(allowedLateness))

and you merge windows that are late, you might end up holding the watermark until the allowedLateness has passed.

Issue Links

• links to: GitHub Pull Request #4793