Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11084

[Go SDK] Validate Windowing

Details

    • Task
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • Not applicable
    • sdk-go
    • None

    Description

      While Windowing basics are  implemented in the Go SDK, there is precious little validation that it's correct. Much work has been done largely for the benefit of Global Windowing.

      DoFn Authors can specify event times for elements by returning a `beam.EventTime` along with their element, allowing windowing transforms by the runner to take effect. They can observe windows in their DoFns by having a `[]beam.Window` parameter, and extract results.

      Pipeline authors can insert `beam.WindowInto` transforms to have elements be batched into the appropriate window slices.

      But very little of this has been validated end to end.  This umbrella task is to test and validate specific aspects of the window implementation to ensure correctness.

      The tricky part is that windows are selected by runners, meaning many of these must be integration tests.

      • Windows are correctly observed in DoFns.
      • Validate windows are aggregated correctly in GBKs and Lifted Combines
        • In particular sliding windows where a single element could contribute to multiple aggregates
      • Validate Side Input behavior WRT windows (implicit observation of windows)
        • In particular, Side Inputs are valid per window.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              lostluck Robert Burke
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 6h 40m
                  6h 40m