[BEAM-696] Document: Side-Inputs non-deterministic with merging main-input windows

• Type: Task
• Status: Resolved
• Priority: P2
• Resolution: Not A Problem
• Fix Version/s: Not applicable
• Component/s: beam-model
• Assignee: Amit Sela
• Reporter: Ben Chambers
• Created: 30/Sep/16 22:58
• Updated: 24/Jul/20 20:55
• Resolved: 30/Mar/17 20:10

Description

Side-Inputs are non-deterministic for several reasons:
1. Because they depend on triggering of the side-input (this is acceptable because triggers are by their nature non-deterministic).
2. They depend on the current state of the main-input window in order to lookup the side-input. This means that with merging
3. Any runner optimizations that affect when the side-input is looked up may cause problems with either or both of these.

This issue focuses on #2 – the non-determinism of side-inputs that execute within a Merging WindowFn.

Possible solution would be to defer running anything that looks up the side-input until we need to extract an output, and using the main-window at that point. Specifically, if the main-window is a MergingWindowFn, don't execute any kind of pre-combine, instead buffer all the inputs and combine later.

This could still run into some non-determinism if there are triggers controlling when we extract output.