Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8657

Not doing Combiner lifting for data-driven triggers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.18.0
    • Component/s: sdk-py-core
    • Labels:
      None

      Description

      The test below will always produce empty output, due to combiner lifting, which already combines all input values of one shard into one before grouping. To fix this, we shall not do Combiner lifting for the data-driven triggers.

      l = [window.TimestampedValue(('a', 1), 1),
      window.TimestampedValue(('b', 3), 3),
      window.TimestampedValue(('a', 2), 2),
      window.TimestampedValue(('a', 5), 5),]

      result = (p | Map(lambda x : x) | 'window' >> beam.WindowInto(FixedWindows(6), trigger=trigger.AfterCount(2), accumulation_mode=trigger.AccumulationMode.DISCARDING) | beam.CombinePerKey(combine.Largest(1)))

        Attachments

          Activity

            People

            • Assignee:
              robertwb Robert Bradshaw
              Reporter:
              liumomo315 wendy liu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h
                2h