Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6105

Inconsistent configuration in property mapreduce.reduce.shuffle.merge.percent

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.4.0, 2.4.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Similar to MAPREDUCE-5906, In MergeManagerImpl.java, the default value of MRJobConfig.SHUFFLE_MERGE_PERCENT(mapreduce.reduce.shuffle.merge.percent) should be 0.66
      According to official doc.
      https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

          this.mergeThreshold = (long)(this.memoryLimit * 
                                jobConf.getFloat(MRJobConfig.SHUFFLE_MERGE_PERCENT, 
                                                 0.90f));
      
      
      1. MAPREDUCE-6105.patch
        3 kB
        Harsh J
      2. MAPREDUCE-6105-01.patch
        1 kB
        Ray Chiang

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2085 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2085/)
        MAPREDUCE-6105. nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2085 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2085/ ) MAPREDUCE-6105 . nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/135/)
        MAPREDUCE-6105. nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/135/ ) MAPREDUCE-6105 . nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2067 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2067/)
        MAPREDUCE-6105. nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2067 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2067/ ) MAPREDUCE-6105 . nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #126 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/126/)
        MAPREDUCE-6105. nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #126 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/126/ ) MAPREDUCE-6105 . nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #135 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/135/)
        MAPREDUCE-6105. nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #135 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/135/ ) MAPREDUCE-6105 . nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #869 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/869/)
        MAPREDUCE-6105. nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #869 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/869/ ) MAPREDUCE-6105 . nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        rchiang Ray Chiang added a comment -

        Thanks for the review and commit!

        Show
        rchiang Ray Chiang added a comment - Thanks for the review and commit!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7336 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7336/)
        MAPREDUCE-6105. nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7336 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7336/ ) MAPREDUCE-6105 . nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. (harsh: rev 685dbafbe2154e5bf4b638da0668ce32d8c879b0) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
        Hide
        qwertymaniac Harsh J added a comment -

        Committed to branch-2 and trunk. Thanks again!

        Show
        qwertymaniac Harsh J added a comment - Committed to branch-2 and trunk. Thanks again!
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12704858/MAPREDUCE-6105.patch
        against trunk revision ce5de93.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5293//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5293//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704858/MAPREDUCE-6105.patch against trunk revision ce5de93. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5293//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5293//console This message is automatically generated.
        Hide
        qwertymaniac Harsh J added a comment -

        +1 for the change, thanks Ray. I just slightly modified your patch to use a constant variable instead of a direct magic number, much like MAPREDUCE-5906.

        Committing after jenkins run.

        Show
        qwertymaniac Harsh J added a comment - +1 for the change, thanks Ray. I just slightly modified your patch to use a constant variable instead of a direct magic number, much like MAPREDUCE-5906 . Committing after jenkins run.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12674914/MAPREDUCE-6105-01.patch
        against trunk revision 0260231.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4963//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4963//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12674914/MAPREDUCE-6105-01.patch against trunk revision 0260231. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4963//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4963//console This message is automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        Submit for testing.

        Show
        rchiang Ray Chiang added a comment - Submit for testing.
        Hide
        rchiang Ray Chiang added a comment -

        Update default value to match documentation.

        Show
        rchiang Ray Chiang added a comment - Update default value to match documentation.

          People

          • Assignee:
            rchiang Ray Chiang
            Reporter:
            dongwook Dongwook Kwon
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development