Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6721

mapreduce.reduce.shuffle.memory.limit.percent=0.0 should be legal to enforce shuffle to disk

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: mrv2, task
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We are potentially hitting an in-memory-shuffle-related reservation starvation resembling MAPREDUCE-6445. To work it around, we wanted to disable in memory shuffle via mapreduce.reduce.shuffle.memory.limit.percent=0.0 that turned out to be disallowed by the current logic. So we had to resort to another small float value such as 0.0001. However, zero is more logical imo.

      1. MAPREDUCE-6721.001.patch
        4 kB
        Gera Shegalov
      2. MAPREDUCE-6721.002.patch
        4 kB
        Gera Shegalov

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user gerashegalov opened a pull request:

          https://github.com/apache/hadoop/pull/102

          MAPREDUCE-6721.001

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/gerashegalov/hadoop feature/MAPREDUCE-6721

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/102.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #102


          commit a41affd11e122eabd6587b92d704fe3c10f315d4
          Author: Gera Shegalov <gera@apache.org>
          Date: 2016-06-22T07:20:35Z

          MAPREDUCE-6721.001


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user gerashegalov opened a pull request: https://github.com/apache/hadoop/pull/102 MAPREDUCE-6721 .001 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gerashegalov/hadoop feature/ MAPREDUCE-6721 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/102.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #102 commit a41affd11e122eabd6587b92d704fe3c10f315d4 Author: Gera Shegalov <gera@apache.org> Date: 2016-06-22T07:20:35Z MAPREDUCE-6721 .001
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Not sure whether github integration works with Jenkins. So attaching the same patch in a usual way.

          Show
          jira.shegalov Gera Shegalov added a comment - Not sure whether github integration works with Jenkins. So attaching the same patch in a usual way.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 54s trunk passed
          +1 compile 0m 29s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 32s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 57s trunk passed
          +1 javadoc 0m 27s trunk passed
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 27s the patch passed
          -1 javac 0m 28s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 157 unchanged - 0 fixed = 158 total (was 157)
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 31s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 7s the patch passed
          +1 javadoc 0m 23s the patch passed
          +1 unit 2m 21s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          17m 17s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Issue MAPREDUCE-6721
          GITHUB PR https://github.com/apache/hadoop/pull/102
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 51916080373b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 17eae9e
          Default Java 1.8.0_91
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6574/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6574/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6574/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 54s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 27s trunk passed +1 mvninstall 0m 29s the patch passed +1 compile 0m 27s the patch passed -1 javac 0m 28s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 1 new + 157 unchanged - 0 fixed = 158 total (was 157) +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 7s the patch passed +1 javadoc 0m 23s the patch passed +1 unit 2m 21s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 17m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Issue MAPREDUCE-6721 GITHUB PR https://github.com/apache/hadoop/pull/102 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 51916080373b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 17eae9e Default Java 1.8.0_91 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6574/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6574/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6574/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          002 to address javac warnings

          Show
          jira.shegalov Gera Shegalov added a comment - 002 to address javac warnings
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 38s trunk passed
          +1 compile 0m 29s trunk passed
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 55s trunk passed
          +1 javadoc 0m 23s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 21s the patch passed
          +1 javac 0m 21s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 26s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 0m 57s the patch passed
          +1 javadoc 0m 20s the patch passed
          -1 unit 2m 4s hadoop-mapreduce-client-core in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          16m 54s



          Reason Tests
          Failed junit tests hadoop.mapreduce.tools.TestCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Issue MAPREDUCE-6721
          GITHUB PR https://github.com/apache/hadoop/pull/102
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux b2f1082f5767 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 17eae9e
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6575/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6575/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6575/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6575/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 38s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 55s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 0m 57s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 2m 4s hadoop-mapreduce-client-core in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 16m 54s Reason Tests Failed junit tests hadoop.mapreduce.tools.TestCLI Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Issue MAPREDUCE-6721 GITHUB PR https://github.com/apache/hadoop/pull/102 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux b2f1082f5767 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 17eae9e Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6575/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6575/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6575/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6575/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob failure is unrelated, tracked in MAPREDUCE-6625

          Show
          jira.shegalov Gera Shegalov added a comment - org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob failure is unrelated, tracked in MAPREDUCE-6625
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Gera Shegalov +1. It's a straight-forward extension of the configuration. Checking this into trunk and branch-2.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Gera Shegalov +1. It's a straight-forward extension of the configuration. Checking this into trunk and branch-2.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/hadoop/pull/102

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/hadoop/pull/102
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Committed to trunk and branch-2.

          Thanks Gera Shegalov for your contribution!

          Show
          ozawa Tsuyoshi Ozawa added a comment - Committed to trunk and branch-2. Thanks Gera Shegalov for your contribution!
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks for a prompt commit Tsuyoshi Ozawa

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks for a prompt commit Tsuyoshi Ozawa
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10001 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10001/)
          MAPREDUCE-6721. mapreduce.reduce.shuffle.memory.limit.percent=0.0 should (ozawa: rev 79a7289165510072c46779251ebb010248cb0ce8)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMergeManager.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10001 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10001/ ) MAPREDUCE-6721 . mapreduce.reduce.shuffle.memory.limit.percent=0.0 should (ozawa: rev 79a7289165510072c46779251ebb010248cb0ce8) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMergeManager.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml

            People

            • Assignee:
              jira.shegalov Gera Shegalov
              Reporter:
              jira.shegalov Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development