Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3304

Go triggering support

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • 2.36.0
    • sdk-go
    • None

    Description

      `Add support for triggers.

      https://beam.apache.org/documentation/programming-guide/#triggers 

      Triggers are special runner side behavior indicating how to handle data WRT the watermark and window. Commonly configuring the processing for “late data” and similar.

      These are not currently implemented for user use in the Go SDK. Reshuffle configures triggers, but it’s not accessible. A correct trigger implementation can at least re-implement Reshuffle in a user pipeline, rather than handled specially within the framework.

       

      Similar to windowing,  many of the same places as https://issues.apache.org/jira/browse/BEAM-11100 need to be modified.

      At simplest though, it's mostly a runner side construction, with less concern on the exec side, and generally much simpler. 

      Appropriate integration tests against portable runners must be implemented:

      https://github.com/apache/beam/tree/master/sdks/go/test/integration/primitives 

      And optionally add support for the configurable triggers to the the Go Direct Runner. However, the results must be compared and validated against a semantically correct runner like the python portable runner first. At minimum, the Go Direct Runner should be made aware of triggers and produce a coherent error whenever there's a trigger it can't deal with.

       

      Attachments

        Issue Links

          Activity

            People

              riteshghorse Ritesh Ghorse
              herohde Henning Rohde
              Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 19h
                  19h