Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7092 MR examples to work better against cloud stores
  3. MAPREDUCE-7091

Terasort on S3A to switch to new committers

          Details

          • Type: Sub-task
          • Status: Resolved
          • Priority: Minor
          • Resolution: Duplicate
          • Affects Version/s: 3.1.0
          • Fix Version/s: 3.3.0
          • Component/s: examples
          • Labels:
            None

            Description

            Terasort is very slow on S3, because it still uses the classic rename-to-commit algorithm on the sort, even while teragen and the reporting can use the new committer

            Reason: org.apache.hadoop.examples.terasort.TeraOutputFormat has overriden getOutputCommitter even though it doesn't need to.

              Attachments

                Issue Links

                Is contained by

                Sub-task - The sub-task of the issue HADOOP-16058 S3A tests to include Terasort

                • Major - Major loss of function.
                • Resolved
                is duplicated by

                Bug - A problem which impairs or prevents the functions of the product. HADOOP-16153 Allow TeraGen to use schema-specific output committer

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved
                is part of

                Sub-task - The sub-task of the issue HADOOP-16058 S3A tests to include Terasort

                • Major - Major loss of function.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      stevel@apache.org Steve Loughran
                      Reporter:
                      stevel@apache.org Steve Loughran
                    • Votes:
                      1 Vote for this issue
                      Watchers:
                      3 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: