Hadoop QA added a comment - 10/May/16 23:39 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 25s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_91 +1 compile 0m 35s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 22s trunk passed +1 javadoc 0m 31s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 37s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_91 +1 javac 0m 36s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_95 +1 javac 0m 34s the patch passed -1 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 4 new + 276 unchanged - 0 fixed = 280 total (was 276) +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 36s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 28s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95 -1 unit 35m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 35m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 91m 29s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.rootMetrics; locked 80% of time Unsynchronized access at FairScheduler.java:80% of time Unsynchronized access at FairScheduler.java: [line 1212] JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803317/YARN-4866.001.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a6a66ff739a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6b1c1cb Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11401/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11401/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Hadoop QA added a comment - 11/May/16 04:51 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 29s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_91 +1 compile 0m 36s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 20s trunk passed +1 javadoc 0m 30s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 37s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_91 +1 javac 0m 35s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_95 +1 javac 0m 33s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 38s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 30s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95 -1 unit 35m 14s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 35m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 91m 37s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.rootMetrics; locked 80% of time Unsynchronized access at FairScheduler.java:80% of time Unsynchronized access at FairScheduler.java: [line 1202] JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803350/YARN-4866.002.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 22ac7429dac5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6e56578 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11405/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11405/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Hadoop QA added a comment - 11/May/16 22:16 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 53s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_91 +1 compile 0m 39s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 29s trunk passed +1 javadoc 0m 36s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 36s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 41s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_91 +1 javac 0m 39s the patch passed +1 compile 0m 37s the patch passed with JDK v1.7.0_95 +1 javac 0m 37s the patch passed -1 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 212 unchanged - 1 fixed = 213 total (was 213) +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 41s the patch passed +1 javadoc 0m 31s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 33s the patch passed with JDK v1.7.0_95 -1 unit 37m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 36m 9s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 96m 14s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803461/YARN-4866.003.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 07584b1920fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / acb509b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11419/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11419/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Yufei Gu added a comment - 12/May/16 17:58 Fixed the style issue.

Yufei Gu added a comment - 13/May/16 16:31 Test failures are unrelated to the patch.

Karthik Kambatla added a comment - 17/May/16 21:45 Thanks for working on this, Yufei. Comments on the patch: In addition to checking the cluster resources, should we also check against queue's max resources for this queue and any parents? For the latter, a helper method might be needed. Nit: Can we avoid naming the variables with a "not" for readability? For instance, may be we could use overMaxAMShareLimit instead of notOverLimit and exhaustsVcores instead of notTakeAllVCore and negate the expressions as necessary? Nit: Also, do we really need the second boolean. How about modifying the newly added if condition to if (!overMaxAMShareLimit && policy is not DRF) and just override overMaxAMShareLimit? The method itself could return !overMaxAMShareLimit? Nit: After the changes, if the code needs more explanation, should we add a comment before the if condition so it is clear why the second check?

Yufei Gu added a comment - 18/May/16 20:56 Thanks Karthik Kambatla for reviewing. I upload a new patch for all suggestions.

Hadoop QA added a comment - 19/May/16 01:48 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 59s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 12s trunk passed +1 javadoc 0m 28s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed -1 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 213 unchanged - 1 fixed = 214 total (was 214) +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 24s the patch passed -1 unit 31m 26s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 49m 38s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Null pointer dereference of queue in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java: [line 523]   Load of known null value in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java: [line 522] Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804785/YARN-4866.005.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 729d8561c284 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 010e6ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11535/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11535/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Hadoop QA added a comment - 19/May/16 06:57 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 32s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 25s trunk passed +1 mvninstall 0m 41s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed -1 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 213 unchanged - 1 fixed = 214 total (was 214) +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 25s the patch passed -1 unit 31m 25s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 47m 3s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Null pointer dereference of queue in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java: [line 523]   Load of known null value in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java: [line 522] Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804785/YARN-4866.005.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9bb14fb371b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11545/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11545/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Yufei Gu added a comment - 19/May/16 23:51 The patch 006 fixed the style issues and the bug, add some test code to test if AM VCores are more than maxResources of queue.

Hadoop QA added a comment - 20/May/16 00:52 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 21s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 29s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 212 unchanged - 1 fixed = 212 total (was 213) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 57s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 29m 31s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 42m 59s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805079/YARN-4866.006.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c5418cd73084 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 42c22f7 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11574/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11574/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11574/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11574/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Yufei Gu added a comment - 20/May/16 01:24 All test failures are unrelated.

Karthik Kambatla added a comment - 24/May/16 18:24 Patch looks close. Shouldn't isVCoreOverMaxResource be added to FSQueue and not FSLeafQueue as a protected method? That way, we don't have to pass the queue. Nit: Javadoc for FSLeafQueue#canRunAppAM mentions Fifo and Fair policy don't care about vcores. I would probably reword and reuse the comment closer to the check - " Check whether this queue can run this application master under the maxAMShare limit. For FIFO and FAIR policies, check if the vcore usage takes up the entire cluster or maxResources for the queue". Would the check for FAIR and FIFO in the method be more readable if we change the order of boolean expressions? overMaxAMShareLimit = isVCoreOverMaxResource(ifRunAMResource) || ifRunAMResource.getVirtualCores() >= scheduler.getRootQueueMetrics().getAvailableVirtualCores();

Yufei Gu added a comment - 25/May/16 00:23 Thanks Karthik Kambatla for the detailed review. 1. The parameter queue is used for recursive checking. It will be there if we put this function in FSQueue. In that case, no obvious reason to move the function to FSQueue, unless FSParentQueue need it in the future. 2. 3. Nice suggestions. Thanks. I've uploaded patch 007 for it.

Hadoop QA added a comment - 25/May/16 01:13 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 18s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 51s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 212 unchanged - 1 fixed = 212 total (was 213) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 10s the patch passed +1 javadoc 0m 22s the patch passed -1 unit 30m 11s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 43m 58s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806031/YARN-4866.007.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 01a4005539dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / edd716e Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11680/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11680/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11680/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11680/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Karthik Kambatla added a comment - 25/May/16 14:21 The parameter queue is used for recursive checking. It will be there if we put this function in FSQueue. In that case, no obvious reason to move the function to FSQueue, unless FSParentQueue need it in the future. Not sure I follow. Would the following not work when implemented in FSQueue? boolean isVcoresOverMaxResource(Resource resource) { if (resource.getVirtualCores() > scheduler.getAllocationConfiguration() .getMaxResources(getName()).getVirtualCores()) { return true ; } FSQueue parent = getParent(); if (parent == null ) { return false ; } return parent().isVcoresOverMaxResource(); }

Yufei Gu added a comment - 25/May/16 16:56 Karthik Kambatla , Got you. Thanks for detailed explanation. Sorry for the misunderstanding. I uploaded patch 008 with the modification you suggested.

Hadoop QA added a comment - 25/May/16 17:49 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 33s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 29s trunk passed +1 javadoc 0m 27s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed -1 checkstyle 0m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 224 unchanged - 1 fixed = 225 total (was 225) +1 mvnsite 0m 43s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 40s the patch passed -1 javadoc 2m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 1255 unchanged - 0 fixed = 1257 total (was 1255) +1 javadoc 0m 27s the patch passed -1 unit 31m 31s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 51m 1s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806161/YARN-4866.008.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 440b1ce85ca8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9a31e5d Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager: https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11686/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11686/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Yufei Gu added a comment - 25/May/16 18:31 Uploaded patch 009 to fix style and javadoc issues.

Hadoop QA added a comment - 25/May/16 19:27 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 53s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 28s the patch passed +1 javac 0m 28s the patch passed +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 223 unchanged - 1 fixed = 223 total (was 224) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 59s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 34m 42s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 48m 41s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806184/YARN-4866.009.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 95e0a79108ce 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 77d5ce9 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11693/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11693/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Karthik Kambatla added a comment - 26/May/16 04:47 +1. Checking this in.

Hudson added a comment - 26/May/16 05:35 SUCCESS: Integrated in Hadoop-trunk-Commit #9864 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9864/ ) YARN-4866 . FairScheduler: AMs can consume all vcores leading to a (kasha: rev 4f513a4a8ed73beec89b7b28c0cd056ee5f4ed0d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java

Karthik Kambatla added a comment - 26/May/16 06:26 Just committed this to trunk and branch-2. Thanks for the contribution, Yufei Gu .

Yufei Gu added a comment - 26/May/16 06:42 Thanks a lot for the review and commit, Karthik Kambatla .