[BEAM-11084] [Go SDK] Validate Windowing

• Type: Task
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-go
• Assignee: Unassigned
• Reporter: Robert Burke
• Created: 21/Oct/20 16:58
• Updated: 07/Mar/22 20:05
• Resolved: 07/Mar/22 20:05

Description

While Windowing basics are  implemented in the Go SDK, there is precious little validation that it's correct. Much work has been done largely for the benefit of Global Windowing.

DoFn Authors can specify event times for elements by returning a `beam.EventTime` along with their element, allowing windowing transforms by the runner to take effect. They can observe windows in their DoFns by having a `[]beam.Window` parameter, and extract results.

Pipeline authors can insert `beam.WindowInto` transforms to have elements be batched into the appropriate window slices.

But very little of this has been validated end to end.  This umbrella task is to test and validate specific aspects of the window implementation to ensure correctness.

The tricky part is that windows are selected by runners, meaning many of these must be integration tests.

• Windows are correctly observed in DoFns.
• Validate windows are aggregated correctly in GBKs and Lifted Combines
  • In particular sliding windows where a single element could contribute to multiple aggregates
• Validate Side Input behavior WRT windows (implicit observation of windows)
  • In particular, Side Inputs are valid per window.

 

Issue Links

• mentioned in: Page Loading...