Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12493

FileIO should allow to opt-in for custom sharding function

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: P2
    • Resolution: Won't Do
    • Affects Version/s: 2.29.0
    • Fix Version/s: None
    • Component/s: sdk-java-core
    • Labels:
      None

      Description

      When number of shards is explicitly specified, then default sharding function is `RandomShardingFunction`. `WriteFiles` does have an option to pass in custom sharding function but that is not surfaced on user facing API at `FileIO`.

      This is limiting in these 2 use-cases:

      1. I need to generate shards which are compatible with Hive bucketing and therefore need to decide shard assignment based on data fields of element being sharded
      2. When run e.g. on Spark and job encounters failure which cause loss of some data from previous stages, Spark does issue recompute of necessary task in necessary stages. Because shard assignment is random, some data will end up in different shards and cause duplicates in final dataset

      I propose to surface `.withShardingFunction()` at FileIO level so user can choose custom sharding strategy when desired.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jvilcek Jozef Vilcek
                Reporter:
                jvilcek Jozef Vilcek
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h