Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6721

setting numshards dynamically for TextIO.write()

Details

    • Improvement
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-java-text
    • None

    Description

      I go through code for TextIO. It uses WriteFiles  in TypedWrite. Now, WriteFiles has the functionality withNumShards(ValueProvider<java.lang.Integer>) and withSharding([PTransform|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/transforms/PTransform.html]<PCollection<UserT>,PCollectionView<java.lang.Integer>> sharding) but TextIO or its inner class TypedWrite doesn't have that. It would really be a great to expose that functionality to user.

      Are there some issues in implementation that this is not already available?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              aditya.guru149 Aditya Guru
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 50m
                  2h 50m