Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4866

FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The maxAMShare uses the queue's policy for enforcing limits. When using FAIR policy, this considers only memory. If there are fewer vcores on the cluster, the AMs can end up taking all the vcores leading to a livelock.

      1. YARN-4866.009.patch
        9 kB
        Yufei Gu
      2. YARN-4866.008.patch
        8 kB
        Yufei Gu
      3. YARN-4866.007.patch
        7 kB
        Yufei Gu
      4. YARN-4866.006.patch
        7 kB
        Yufei Gu
      5. YARN-4866.005.patch
        6 kB
        Yufei Gu
      6. YARN-4866.004.patch
        6 kB
        Yufei Gu
      7. YARN-4866.003.patch
        6 kB
        Yufei Gu
      8. YARN-4866.002.patch
        5 kB
        Yufei Gu
      9. YARN-4866.001.patch
        5 kB
        Yufei Gu

        Issue Links

          Activity

          Hide
          yufeigu Yufei Gu added a comment -

          Thanks a lot for the review and commit, Karthik Kambatla.

          Show
          yufeigu Yufei Gu added a comment - Thanks a lot for the review and commit, Karthik Kambatla .
          Hide
          kasha Karthik Kambatla added a comment -

          Just committed this to trunk and branch-2.

          Thanks for the contribution, Yufei Gu.

          Show
          kasha Karthik Kambatla added a comment - Just committed this to trunk and branch-2. Thanks for the contribution, Yufei Gu .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9864 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9864/)
          YARN-4866. FairScheduler: AMs can consume all vcores leading to a (kasha: rev 4f513a4a8ed73beec89b7b28c0cd056ee5f4ed0d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9864 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9864/ ) YARN-4866 . FairScheduler: AMs can consume all vcores leading to a (kasha: rev 4f513a4a8ed73beec89b7b28c0cd056ee5f4ed0d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Hide
          kasha Karthik Kambatla added a comment -

          +1. Checking this in.

          Show
          kasha Karthik Kambatla added a comment - +1. Checking this in.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 46s trunk passed
          +1 compile 0m 29s trunk passed
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 32s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 53s trunk passed
          +1 javadoc 0m 22s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 28s the patch passed
          +1 javac 0m 28s the patch passed
          +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 223 unchanged - 1 fixed = 223 total (was 224)
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 59s the patch passed
          +1 javadoc 0m 19s the patch passed
          -1 unit 34m 42s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 15s Patch does not generate ASF License warnings.
          48m 41s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806184/YARN-4866.009.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 95e0a79108ce 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 77d5ce9
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11693/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11693/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 53s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 28s the patch passed +1 javac 0m 28s the patch passed +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 223 unchanged - 1 fixed = 223 total (was 224) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 59s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 34m 42s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 48m 41s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806184/YARN-4866.009.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 95e0a79108ce 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 77d5ce9 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11693/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11693/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11693/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Uploaded patch 009 to fix style and javadoc issues.

          Show
          yufeigu Yufei Gu added a comment - Uploaded patch 009 to fix style and javadoc issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 33s trunk passed
          +1 compile 0m 37s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 45s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 29s trunk passed
          +1 javadoc 0m 27s trunk passed
          +1 mvninstall 0m 38s the patch passed
          +1 compile 0m 37s the patch passed
          +1 javac 0m 37s the patch passed
          -1 checkstyle 0m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 224 unchanged - 1 fixed = 225 total (was 225)
          +1 mvnsite 0m 43s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 40s the patch passed
          -1 javadoc 2m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 1255 unchanged - 0 fixed = 1257 total (was 1255)
          +1 javadoc 0m 27s the patch passed
          -1 unit 31m 31s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 16s Patch does not generate ASF License warnings.
          51m 1s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806161/YARN-4866.008.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 440b1ce85ca8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9a31e5d
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager: https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11686/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11686/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 33s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 29s trunk passed +1 javadoc 0m 27s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed -1 checkstyle 0m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 224 unchanged - 1 fixed = 225 total (was 225) +1 mvnsite 0m 43s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 40s the patch passed -1 javadoc 2m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 1255 unchanged - 0 fixed = 1257 total (was 1255) +1 javadoc 0m 27s the patch passed -1 unit 31m 31s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 51m 1s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806161/YARN-4866.008.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 440b1ce85ca8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9a31e5d Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager: https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11686/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11686/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Karthik Kambatla, Got you. Thanks for detailed explanation. Sorry for the misunderstanding. I uploaded patch 008 with the modification you suggested.

          Show
          yufeigu Yufei Gu added a comment - Karthik Kambatla , Got you. Thanks for detailed explanation. Sorry for the misunderstanding. I uploaded patch 008 with the modification you suggested.
          Hide
          kasha Karthik Kambatla added a comment -

          The parameter queue is used for recursive checking. It will be there if we put this function in FSQueue. In that case, no obvious reason to move the function to FSQueue, unless FSParentQueue need it in the future.

          Not sure I follow. Would the following not work when implemented in FSQueue?

          boolean isVcoresOverMaxResource(Resource resource) {
            if (resource.getVirtualCores() > 
                  scheduler.getAllocationConfiguration()
                       .getMaxResources(getName()).getVirtualCores()) {
               return true;
            }
          
            FSQueue parent = getParent();
            if (parent == null) {
              return false;
            }
          
            return parent().isVcoresOverMaxResource();
          }
          
          Show
          kasha Karthik Kambatla added a comment - The parameter queue is used for recursive checking. It will be there if we put this function in FSQueue. In that case, no obvious reason to move the function to FSQueue, unless FSParentQueue need it in the future. Not sure I follow. Would the following not work when implemented in FSQueue? boolean isVcoresOverMaxResource(Resource resource) { if (resource.getVirtualCores() > scheduler.getAllocationConfiguration() .getMaxResources(getName()).getVirtualCores()) { return true ; } FSQueue parent = getParent(); if (parent == null ) { return false ; } return parent().isVcoresOverMaxResource(); }
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 18s trunk passed
          +1 compile 0m 28s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 32s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 51s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 212 unchanged - 1 fixed = 212 total (was 213)
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 10s the patch passed
          +1 javadoc 0m 22s the patch passed
          -1 unit 30m 11s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          43m 58s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806031/YARN-4866.007.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 01a4005539dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / edd716e
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11680/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11680/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11680/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11680/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 18s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 51s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 212 unchanged - 1 fixed = 212 total (was 213) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 10s the patch passed +1 javadoc 0m 22s the patch passed -1 unit 30m 11s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 43m 58s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806031/YARN-4866.007.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 01a4005539dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / edd716e Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11680/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11680/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11680/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11680/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Karthik Kambatla for the detailed review.

          1. The parameter queue is used for recursive checking. It will be there if we put this function in FSQueue. In that case, no obvious reason to move the function to FSQueue, unless FSParentQueue need it in the future.
          2. 3. Nice suggestions. Thanks. I've uploaded patch 007 for it.

          Show
          yufeigu Yufei Gu added a comment - Thanks Karthik Kambatla for the detailed review. 1. The parameter queue is used for recursive checking. It will be there if we put this function in FSQueue. In that case, no obvious reason to move the function to FSQueue, unless FSParentQueue need it in the future. 2. 3. Nice suggestions. Thanks. I've uploaded patch 007 for it.
          Hide
          kasha Karthik Kambatla added a comment -

          Patch looks close.

          1. Shouldn't isVCoreOverMaxResource be added to FSQueue and not FSLeafQueue as a protected method? That way, we don't have to pass the queue.
          2. Nit: Javadoc for FSLeafQueue#canRunAppAM mentions Fifo and Fair policy don't care about vcores. I would probably reword and reuse the comment closer to the check - " Check whether this queue can run this application master under the maxAMShare limit. For FIFO and FAIR policies, check if the vcore usage takes up the entire cluster or maxResources for the queue".
          3. Would the check for FAIR and FIFO in the method be more readable if we change the order of boolean expressions?
            overMaxAMShareLimit = isVCoreOverMaxResource(ifRunAMResource) ||
                  ifRunAMResource.getVirtualCores() >= 
                      scheduler.getRootQueueMetrics().getAvailableVirtualCores();
            
          Show
          kasha Karthik Kambatla added a comment - Patch looks close. Shouldn't isVCoreOverMaxResource be added to FSQueue and not FSLeafQueue as a protected method? That way, we don't have to pass the queue. Nit: Javadoc for FSLeafQueue#canRunAppAM mentions Fifo and Fair policy don't care about vcores. I would probably reword and reuse the comment closer to the check - " Check whether this queue can run this application master under the maxAMShare limit. For FIFO and FAIR policies, check if the vcore usage takes up the entire cluster or maxResources for the queue". Would the check for FAIR and FIFO in the method be more readable if we change the order of boolean expressions? overMaxAMShareLimit = isVCoreOverMaxResource(ifRunAMResource) || ifRunAMResource.getVirtualCores() >= scheduler.getRootQueueMetrics().getAvailableVirtualCores();
          Hide
          yufeigu Yufei Gu added a comment -

          All test failures are unrelated.

          Show
          yufeigu Yufei Gu added a comment - All test failures are unrelated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 21s trunk passed
          +1 compile 0m 30s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 33s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 56s trunk passed
          +1 javadoc 0m 22s trunk passed
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 26s the patch passed
          +1 javac 0m 26s the patch passed
          +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 212 unchanged - 1 fixed = 212 total (was 213)
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 57s the patch passed
          +1 javadoc 0m 19s the patch passed
          -1 unit 29m 31s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          42m 59s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805079/YARN-4866.006.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c5418cd73084 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 42c22f7
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11574/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11574/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11574/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11574/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 21s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 29s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 0 new + 212 unchanged - 1 fixed = 212 total (was 213) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 57s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 29m 31s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 42m 59s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805079/YARN-4866.006.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c5418cd73084 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 42c22f7 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11574/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11574/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11574/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11574/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          The patch 006 fixed the style issues and the bug, add some test code to test if AM VCores are more than maxResources of queue.

          Show
          yufeigu Yufei Gu added a comment - The patch 006 fixed the style issues and the bug, add some test code to test if AM VCores are more than maxResources of queue.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 32s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 9s trunk passed
          +1 javadoc 0m 25s trunk passed
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 37s the patch passed
          +1 javac 0m 37s the patch passed
          -1 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 213 unchanged - 1 fixed = 214 total (was 214)
          +1 mvnsite 0m 41s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 25s the patch passed
          -1 unit 31m 25s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 15s Patch does not generate ASF License warnings.
          47m 3s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Null pointer dereference of queue in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java:[line 523]
            Load of known null value in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java:[line 522]
          Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804785/YARN-4866.005.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9bb14fb371b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1597630
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11545/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11545/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 32s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 25s trunk passed +1 mvninstall 0m 41s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed -1 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 213 unchanged - 1 fixed = 214 total (was 214) +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 25s the patch passed -1 unit 31m 25s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 47m 3s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Null pointer dereference of queue in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java: [line 523]   Load of known null value in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java: [line 522] Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804785/YARN-4866.005.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9bb14fb371b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11545/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11545/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 59s trunk passed
          +1 compile 0m 35s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 45s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 28s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 34s the patch passed
          +1 javac 0m 34s the patch passed
          -1 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 213 unchanged - 1 fixed = 214 total (was 214)
          +1 mvnsite 0m 41s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 24s the patch passed
          -1 unit 31m 26s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          49m 38s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Null pointer dereference of queue in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java:[line 523]
            Load of known null value in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java:[line 522]
          Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804785/YARN-4866.005.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 729d8561c284 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 010e6ac
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11535/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11535/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 59s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 12s trunk passed +1 javadoc 0m 28s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed -1 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 213 unchanged - 1 fixed = 214 total (was 214) +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 24s the patch passed -1 unit 31m 26s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 49m 38s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Null pointer dereference of queue in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) Dereferenced at FSLeafQueue.java: [line 523]   Load of known null value in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java:in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource, FSQueue) At FSLeafQueue.java: [line 522] Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804785/YARN-4866.005.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 729d8561c284 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 010e6ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11535/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11535/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11535/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Karthik Kambatla for reviewing.

          I upload a new patch for all suggestions.

          Show
          yufeigu Yufei Gu added a comment - Thanks Karthik Kambatla for reviewing. I upload a new patch for all suggestions.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for working on this, Yufei. Comments on the patch:

          1. In addition to checking the cluster resources, should we also check against queue's max resources for this queue and any parents? For the latter, a helper method might be needed.
          2. Nit: Can we avoid naming the variables with a "not" for readability? For instance, may be we could use overMaxAMShareLimit instead of notOverLimit and exhaustsVcores instead of notTakeAllVCore and negate the expressions as necessary?
          3. Nit: Also, do we really need the second boolean. How about modifying the newly added if condition to if (!overMaxAMShareLimit && policy is not DRF) and just override overMaxAMShareLimit? The method itself could return !overMaxAMShareLimit?
          4. Nit: After the changes, if the code needs more explanation, should we add a comment before the if condition so it is clear why the second check?
          Show
          kasha Karthik Kambatla added a comment - Thanks for working on this, Yufei. Comments on the patch: In addition to checking the cluster resources, should we also check against queue's max resources for this queue and any parents? For the latter, a helper method might be needed. Nit: Can we avoid naming the variables with a "not" for readability? For instance, may be we could use overMaxAMShareLimit instead of notOverLimit and exhaustsVcores instead of notTakeAllVCore and negate the expressions as necessary? Nit: Also, do we really need the second boolean. How about modifying the newly added if condition to if (!overMaxAMShareLimit && policy is not DRF) and just override overMaxAMShareLimit? The method itself could return !overMaxAMShareLimit? Nit: After the changes, if the code needs more explanation, should we add a comment before the if condition so it is clear why the second check?
          Hide
          yufeigu Yufei Gu added a comment -

          Test failures are unrelated to the patch.

          Show
          yufeigu Yufei Gu added a comment - Test failures are unrelated to the patch.
          Hide
          yufeigu Yufei Gu added a comment -

          Fixed the style issue.

          Show
          yufeigu Yufei Gu added a comment - Fixed the style issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 53s trunk passed
          +1 compile 0m 39s trunk passed with JDK v1.8.0_91
          +1 compile 0m 39s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 45s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 29s trunk passed
          +1 javadoc 0m 36s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 36s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_91
          +1 javac 0m 39s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.7.0_95
          +1 javac 0m 37s the patch passed
          -1 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 212 unchanged - 1 fixed = 213 total (was 213)
          +1 mvnsite 0m 41s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 41s the patch passed
          +1 javadoc 0m 31s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 33s the patch passed with JDK v1.7.0_95
          -1 unit 37m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          -1 unit 36m 9s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          96m 14s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803461/YARN-4866.003.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 07584b1920fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / acb509b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11419/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11419/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 53s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_91 +1 compile 0m 39s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 29s trunk passed +1 javadoc 0m 36s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 36s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 41s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_91 +1 javac 0m 39s the patch passed +1 compile 0m 37s the patch passed with JDK v1.7.0_95 +1 javac 0m 37s the patch passed -1 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 212 unchanged - 1 fixed = 213 total (was 213) +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 41s the patch passed +1 javadoc 0m 31s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 33s the patch passed with JDK v1.7.0_95 -1 unit 37m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 36m 9s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 96m 14s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803461/YARN-4866.003.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 07584b1920fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / acb509b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11419/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11419/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11419/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 29s trunk passed
          +1 compile 0m 37s trunk passed with JDK v1.8.0_91
          +1 compile 0m 36s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 20s trunk passed
          +1 javadoc 0m 30s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.8.0_91
          +1 javac 0m 35s the patch passed
          +1 compile 0m 33s the patch passed with JDK v1.7.0_95
          +1 javac 0m 33s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 0m 41s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 38s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 30s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95
          -1 unit 35m 14s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          -1 unit 35m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          91m 37s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.rootMetrics; locked 80% of time Unsynchronized access at FairScheduler.java:80% of time Unsynchronized access at FairScheduler.java:[line 1202]
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803350/YARN-4866.002.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 22ac7429dac5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6e56578
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11405/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11405/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 29s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_91 +1 compile 0m 36s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 20s trunk passed +1 javadoc 0m 30s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 37s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_91 +1 javac 0m 35s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_95 +1 javac 0m 33s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 38s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 30s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95 -1 unit 35m 14s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 35m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 91m 37s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.rootMetrics; locked 80% of time Unsynchronized access at FairScheduler.java:80% of time Unsynchronized access at FairScheduler.java: [line 1202] JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803350/YARN-4866.002.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 22ac7429dac5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6e56578 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11405/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11405/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11405/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 25s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_91
          +1 compile 0m 35s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 22s trunk passed
          +1 javadoc 0m 31s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.8.0_91
          +1 javac 0m 36s the patch passed
          +1 compile 0m 34s the patch passed with JDK v1.7.0_95
          +1 javac 0m 34s the patch passed
          -1 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 4 new + 276 unchanged - 0 fixed = 280 total (was 276)
          +1 mvnsite 0m 40s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 36s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 28s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95
          -1 unit 35m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          -1 unit 35m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          91m 29s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.rootMetrics; locked 80% of time Unsynchronized access at FairScheduler.java:80% of time Unsynchronized access at FairScheduler.java:[line 1212]
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803317/YARN-4866.001.patch
          JIRA Issue YARN-4866
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a6a66ff739a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6b1c1cb
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11401/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11401/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 25s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_91 +1 compile 0m 35s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 22s trunk passed +1 javadoc 0m 31s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 32s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 37s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_91 +1 javac 0m 36s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_95 +1 javac 0m 34s the patch passed -1 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 4 new + 276 unchanged - 0 fixed = 280 total (was 276) +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 36s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 28s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 30s the patch passed with JDK v1.7.0_95 -1 unit 35m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 35m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 91m 29s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.rootMetrics; locked 80% of time Unsynchronized access at FairScheduler.java:80% of time Unsynchronized access at FairScheduler.java: [line 1212] JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803317/YARN-4866.001.patch JIRA Issue YARN-4866 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a6a66ff739a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6b1c1cb Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11401/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11401/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11401/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

            People

            • Assignee:
              yufeigu Yufei Gu
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development