Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12040

WriteFiles withRunnerDeterminedSharding for unbounded data doesn't work with merging windows

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-java-files
    • None

    Description

      Currently the implementation of `withRunnerDeterminedShardingUnbounded` uses a stateful DoFn to achieve the grouping and batching of the input elements, which doesn't support session windows. One possible way is to add another GBK prior to the stateful DoFn to first get session windows merged and reify the window before invoking the sateful DoFn. 

      Attachments

        Activity

          People

            Unassigned Unassigned
            sychen Siyuan Chen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 50m
                5h 50m