Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7092 MR examples to work better against cloud stores
  3. MAPREDUCE-7091

Terasort on S3A to switch to new committers

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 3.1.0
    • 3.3.0
    • examples
    • None

    Description

      Terasort is very slow on S3, because it still uses the classic rename-to-commit algorithm on the sort, even while teragen and the reporting can use the new committer

      Reason: org.apache.hadoop.examples.terasort.TeraOutputFormat has overriden getOutputCommitter even though it doesn't need to.

      Attachments

        Issue Links

          Is contained by

          Sub-task - The sub-task of the issue HADOOP-16058 S3A tests to include Terasort

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16153 Allow TeraGen to use schema-specific output committer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is part of

          Sub-task - The sub-task of the issue HADOOP-16058 S3A tests to include Terasort

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: