Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12856

Allow for configuration of unbounded reader max elements, read time etc in StreamingDataflowRunner

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.34.0
    • runner-dataflow
    • None

    Description

      Currently in WorkerCustomSources.java it is hard-coded to 10 seconds, 10k elements, with 1sec waiting for elements if none are available.
      There are cases where it would be beneficial to wait longer and process more data so it would be nice if this was controlled by pipeline option.

      Attachments

        Issue Links

          Activity

            People

              scwhittle Sam Whittle
              scwhittle Sam Whittle
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h
                  2h