Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6765

MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: mr-am
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When mapper or reducer containers request resource larger than the maxResourceRequest in the cluster, job is to be killed. In such cases, it is unnecessary to still schedule container requests.

      1. mapreduce6765.005.patch
        15 kB
        Haibo Chen
      2. mapreduce6765.004.patch
        15 kB
        Haibo Chen
      3. mapreduce6765.003.patch
        15 kB
        Haibo Chen
      4. mapreduce6765.002.patch
        3 kB
        Haibo Chen
      5. mapreduce6765.001.patch
        1 kB
        Haibo Chen

        Activity

        Hide
        haibochen Haibo Chen added a comment -

        Not a 100% sure, it seems that you'll have to be added by a committer to become a contributor first.

        Show
        haibochen Haibo Chen added a comment - Not a 100% sure, it seems that you'll have to be added by a committer to become a contributor first.
        Hide
        haibochen Haibo Chen added a comment -

        The patch is trivial. In cases where the resource requested by container is not supported, do not scheduler container requests, since the job will soon be killed.

        There is no test since the fix is trivial, and ScheduledRequests is an inner class whose ScheduledRequests.addMap or ScheduledRequests.addReduce are hard to spy on.

        Show
        haibochen Haibo Chen added a comment - The patch is trivial. In cases where the resource requested by container is not supported, do not scheduler container requests, since the job will soon be killed. There is no test since the fix is trivial, and ScheduledRequests is an inner class whose ScheduledRequests.addMap or ScheduledRequests.addReduce are hard to spy on.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 46s trunk passed
        +1 compile 0m 23s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 39s trunk passed
        +1 javadoc 0m 17s trunk passed
        +1 mvninstall 0m 21s the patch passed
        +1 compile 0m 20s the patch passed
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 41s the patch passed
        +1 javadoc 0m 13s the patch passed
        +1 unit 8m 55s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        22m 38s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825319/mapreduce6765.001.patch
        JIRA Issue MAPREDUCE-6765
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 69fa0eb9733c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3476156
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6694/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6694/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 46s trunk passed +1 compile 0m 23s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 39s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 8m 55s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 22m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825319/mapreduce6765.001.patch JIRA Issue MAPREDUCE-6765 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 69fa0eb9733c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3476156 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6694/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6694/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        haibochen Haibo Chen added a comment - - edited

        Tested it on a pseudo-distributed cluster, ran a pi job that requests a container bigger than what is supported at maximum, worked fine.

        Show
        haibochen Haibo Chen added a comment - - edited Tested it on a pseudo-distributed cluster, ran a pi job that requests a container bigger than what is supported at maximum, worked fine.
        Hide
        templedf Daniel Templeton added a comment -

        I share Djikstra's dislike of multiple exit points from a function, especially in this case where the code is explicitly structured as a series of conditionals to avoid having multiple exit points. In both cases, could you please put the subsequent few lines of code in an else instead of adding the returns?

        Show
        templedf Daniel Templeton added a comment - I share Djikstra's dislike of multiple exit points from a function, especially in this case where the code is explicitly structured as a series of conditionals to avoid having multiple exit points. In both cases, could you please put the subsequent few lines of code in an else instead of adding the returns?
        Hide
        haibochen Haibo Chen added a comment -

        Updating a new patch to address Daniel's comment.

        Show
        haibochen Haibo Chen added a comment - Updating a new patch to address Daniel's comment.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 45s trunk passed
        +1 compile 0m 26s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 36s trunk passed
        +1 javadoc 0m 16s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 20s the patch passed
        +1 javac 0m 20s the patch passed
        -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 3 new + 94 unchanged - 5 fixed = 97 total (was 99)
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 39s the patch passed
        +1 javadoc 0m 13s the patch passed
        -1 unit 8m 38s hadoop-mapreduce-client-app in the patch failed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        21m 19s



        Reason Tests
        Failed junit tests hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828789/mapreduce6765.002.patch
        JIRA Issue MAPREDUCE-6765
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 827fbe00473f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ec3ea18
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 45s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 36s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -1 checkstyle 0m 14s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 3 new + 94 unchanged - 5 fixed = 97 total (was 99) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 13s the patch passed -1 unit 8m 38s hadoop-mapreduce-client-app in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 21m 19s Reason Tests Failed junit tests hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828789/mapreduce6765.002.patch JIRA Issue MAPREDUCE-6765 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 827fbe00473f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ec3ea18 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6722/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        LGTM. +1 (non-binding)

        Show
        templedf Daniel Templeton added a comment - LGTM. +1 (non-binding)
        Hide
        rkanter Robert Kanter added a comment -

        Haibo Chen, the test failures look related given that the only changes were in RMContainerAllocator and the test failures were all in TestRMContainerAllocator. Can you take a look?

        Show
        rkanter Robert Kanter added a comment - Haibo Chen , the test failures look related given that the only changes were in RMContainerAllocator and the test failures were all in TestRMContainerAllocator . Can you take a look?
        Hide
        haibochen Haibo Chen added a comment -

        Sorry for not checking this closely. I missed the outer if statement (ResourceRequest.equals(Resources.none()). I will upload a new patch to address the failures.

        Show
        haibochen Haibo Chen added a comment - Sorry for not checking this closely. I missed the outer if statement (ResourceRequest.equals(Resources.none()). I will upload a new patch to address the failures.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 16s trunk passed
        +1 compile 0m 30s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 37s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 0m 49s trunk passed
        +1 javadoc 0m 20s trunk passed
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 28s the patch passed
        -1 javac 0m 28s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app generated 3 new + 10 unchanged - 0 fixed = 13 total (was 10)
        -1 checkstyle 0m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 4 new + 210 unchanged - 16 fixed = 214 total (was 226)
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 1s the patch passed
        +1 javadoc 0m 20s the patch passed
        +1 unit 8m 56s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        26m 4s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830232/mapreduce6765.003.patch
        JIRA Issue MAPREDUCE-6765
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 13cb7687cb79 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5707f88
        Default Java 1.8.0_101
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6737/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6737/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6737/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6737/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 16s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 28s the patch passed -1 javac 0m 28s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app generated 3 new + 10 unchanged - 0 fixed = 13 total (was 10) -1 checkstyle 0m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 4 new + 210 unchanged - 16 fixed = 214 total (was 226) +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 20s the patch passed +1 unit 8m 56s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 26m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830232/mapreduce6765.003.patch JIRA Issue MAPREDUCE-6765 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 13cb7687cb79 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5707f88 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6737/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6737/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6737/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6737/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        haibochen Haibo Chen added a comment -

        New patch to fix checkstyle warnings

        Show
        haibochen Haibo Chen added a comment - New patch to fix checkstyle warnings
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 56s trunk passed
        +1 compile 0m 25s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 39s trunk passed
        +1 javadoc 0m 16s trunk passed
        +1 mvninstall 0m 21s the patch passed
        +1 compile 0m 21s the patch passed
        +1 javac 0m 21s the patch passed
        +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 0 new + 210 unchanged - 16 fixed = 210 total (was 226)
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 41s the patch passed
        +1 javadoc 0m 13s the patch passed
        +1 unit 8m 53s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        22m 54s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830364/mapreduce6765.004.patch
        JIRA Issue MAPREDUCE-6765
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8cc811d11350 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4815d02
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6740/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6740/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 56s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 39s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 0 new + 210 unchanged - 16 fixed = 210 total (was 226) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 8m 53s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 22m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830364/mapreduce6765.004.patch JIRA Issue MAPREDUCE-6765 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8cc811d11350 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4815d02 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6740/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6740/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment -

        It overall looks good to me. But is there anyway to reduce the amount of duplicate code? The handleMapContainerRequest and handleReduceContainerRequest methods are very similar.

        Show
        rkanter Robert Kanter added a comment - It overall looks good to me. But is there anyway to reduce the amount of duplicate code? The handleMapContainerRequest and handleReduceContainerRequest methods are very similar.
        Hide
        haibochen Haibo Chen added a comment -

        They are very similar in terms of structure. But not sure how much we can cut down the duplicate code. The log message, functions on pendingReduces and scheduledRequests are different.

        Show
        haibochen Haibo Chen added a comment - They are very similar in terms of structure. But not sure how much we can cut down the duplicate code. The log message, functions on pendingReduces and scheduledRequests are different.
        Hide
        rkanter Robert Kanter added a comment -

        +1

        Show
        rkanter Robert Kanter added a comment - +1
        Hide
        rkanter Robert Kanter added a comment -

        Haibo Chen, the patch no longer cleanly applies; can you rebase the patch?

        Show
        rkanter Robert Kanter added a comment - Haibo Chen , the patch no longer cleanly applies; can you rebase the patch?
        Hide
        haibochen Haibo Chen added a comment -

        Uploading a new patch that applies to trunk cleanly.

        Show
        haibochen Haibo Chen added a comment - Uploading a new patch that applies to trunk cleanly.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 39s trunk passed
        +1 compile 0m 25s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 31s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 39s trunk passed
        +1 javadoc 0m 16s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 22s the patch passed
        +1 javac 0m 22s the patch passed
        +1 checkstyle 0m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 0 new + 210 unchanged - 16 fixed = 210 total (was 226)
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 48s the patch passed
        +1 javadoc 0m 16s the patch passed
        +1 unit 9m 37s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        24m 44s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835450/mapreduce6765.005.patch
        JIRA Issue MAPREDUCE-6765
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bf7854d76971 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 22ff0ef
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6774/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6774/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 39s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 39s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 22s the patch passed +1 javac 0m 22s the patch passed +1 checkstyle 0m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 0 new + 210 unchanged - 16 fixed = 210 total (was 226) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 48s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 9m 37s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 24m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835450/mapreduce6765.005.patch JIRA Issue MAPREDUCE-6765 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bf7854d76971 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22ff0ef Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6774/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6774/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment -

        Thanks Haibo Chen. Committed to trunk and branch-2!

        Show
        rkanter Robert Kanter added a comment - Thanks Haibo Chen . Committed to trunk and branch-2!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10751 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10751/)
        MAPREDUCE-6765. MR should not schedule container requests in cases where (rkanter: rev fc2b69eba1c5df59f6175205c27dc7b584df50c0)

        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10751 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10751/ ) MAPREDUCE-6765 . MR should not schedule container requests in cases where (rkanter: rev fc2b69eba1c5df59f6175205c27dc7b584df50c0) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java

          People

          • Assignee:
            haibochen Haibo Chen
            Reporter:
            haibochen Haibo Chen
          • Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development