Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7092 MR examples to work better against cloud stores
  3. MAPREDUCE-7091

Terasort on S3A to switch to new committers

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.3.0
    • Component/s: examples
    • Labels:
      None

      Description

      Terasort is very slow on S3, because it still uses the classic rename-to-commit algorithm on the sort, even while teragen and the reporting can use the new committer

      Reason: org.apache.hadoop.examples.terasort.TeraOutputFormat has overriden getOutputCommitter even though it doesn't need to.

        Attachments

          Issue Links

          Is contained by

          Sub-task - The sub-task of the issue HADOOP-16058 S3A tests to include Terasort

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16153 Allow TeraGen to use schema-specific output committer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is part of

          Sub-task - The sub-task of the issue HADOOP-16058 S3A tests to include Terasort

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: