Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6721

setting numshards dynamically for TextIO.write()

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: io-java-text
    • Labels:
      None

      Description

      I go through code for TextIO. It uses WriteFiles  in TypedWrite. Now, WriteFiles has the functionality withNumShards(ValueProvider<java.lang.Integer>) and withSharding([PTransform|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/transforms/PTransform.html]<PCollection<UserT>,PCollectionView<java.lang.Integer>> sharding) but TextIO or its inner class TypedWrite doesn't have that. It would really be a great to expose that functionality to user.

      Are there some issues in implementation that this is not already available?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              aditya.guru149 Aditya Guru
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: