Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7268

Make external sorter Hadoop free

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.13.0
    • Fix Version/s: 2.14.0
    • Component/s: extensions-java-sorter
    • Labels:
      None

      Description

      Right now the Java sorter extension depends on Hadoop SequenceFile for external sort. It'll be nice to re-implement it without the dependency to avoid conflicts.

      Pull request adds a new option to switch to native sorting, the default behavior remains Hadoop sorting.

          final BufferedExternalSorter.Options sorterOpts =
              BufferedExternalSorter
                  .options()
                  .withExternalSorterType(ExternalSorter.Options.SorterType.HADOOP);
      
      ...
      
              .apply(SortValues.create(sorterOpts));
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sinisa_lyh Neville Li
                Reporter:
                sinisa_lyh Neville Li
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h