Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11084 [Go SDK] Validate Windowing
  3. BEAM-11087

[Go SDK] Validate Side Input behavior WRT windows

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • 2.35.0
    • sdk-go
    • None

    Description

      DoFns with Side inputs implicitly observe windows, as Side Inputs are scoped to a current window, a powerful feature of beam.

      The ideal would be to you could try to unit test this in the exec package, creating a fake side input adapter (or use the real one), to more directly target the implementation. 

      https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/sideinput.go#L34 and exercising the ParDo code for Side Input handling directly.

      https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/pardo.go#L38 
      Then it should be possible to bolster that with appropriate  mock windows to ensure that side inputs are configured correctly.

      Altnernatively this behavior could be tested and validated with an integration test against real runners due to the complexity around Side Inputs making unit testing a challenge. (While side input code could be tested in that fashion, it's likely dramatically simpler to do the integration test.)

      Some light tests with Side Inputs already exist, but they're purely in the Global Window. Add tests for non-global windows to ensure that Side Inputs are scoped correctly.

      Integration test directory: https://github.com/apache/beam/tree/master/sdks/go/test/integration/primitives

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jrmccluskey Jack McCluskey
            lostluck Robert Burke
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 20m
                3h 20m

                Issue deployment