Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12685

Allow managed thread count in AvroIO

Details

    • Improvement
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-java-avro, io-java-files
    • None

    Description

      During execution, the `ReadAllViaFileBasedSource` runs ReShuffle and creates an un-grouped set of file range readers.  This can easily cause OOM issues when the number of groups changes as there is no limit to the number of concurrent file reads.

       

      Using Reshuffle.viaRandomKeys.withNumBuckets instead will allow the same default behavior, but lets the user configure the number of readers as and when needed.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              dhercher Dylan Hercher

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 2h Original Estimate - 2h
                  2h
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h

                  Slack

                    Issue deployment