Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2306

Add test for leakage of reservation metrics in fair scheduler

    Details

    • Hadoop Flags:
      Reviewed

      Description

      This only applies to fair scheduler. Capacity scheduler is OK.
      When appAttempt or node is removed, the metrics for reservation(reservedContainers, reservedMB, reservedVCores) is not reduced back.

      These are important metrics for administrator. The wrong metrics confuses may confuse them.

      1. YARN-2306.004.patch
        3 kB
        Yufei Gu
      2. YARN-2306.patch
        4 kB
        Hong Zhiguo
      3. YARN-2306-2.patch
        3 kB
        Hong Zhiguo
      4. YARN-2306-3.patch
        3 kB
        Hong Zhiguo

        Activity

        Hide
        zhiguohong Hong Zhiguo added a comment -

        fix with new unit test TestFairScheduler#testReservationMetrics to reveal this bug.

        Show
        zhiguohong Hong Zhiguo added a comment - fix with new unit test TestFairScheduler#testReservationMetrics to reveal this bug.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12656254/YARN-2306.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4342//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4342//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656254/YARN-2306.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4342//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4342//console This message is automatically generated.
        Hide
        ywskycn Wei Yan added a comment -

        Thanks, Hong Zhiguo, the patch looks good to me.

        Show
        ywskycn Wei Yan added a comment - Thanks, Hong Zhiguo , the patch looks good to me.
        Hide
        ashwinshankar77 Ashwin Shankar added a comment -

        Karthik Kambatla, we saw reservation metrics leak in our clusters as well.
        If this patch looks good to you, can we please commit it ?

        Show
        ashwinshankar77 Ashwin Shankar added a comment - Karthik Kambatla , we saw reservation metrics leak in our clusters as well. If this patch looks good to you, can we please commit it ?
        Hide
        kkambatl Karthik Kambatla (Inactive) added a comment -

        The patch doesn't apply to trunk anymore. Can someone update the patch? Thanks.

        Show
        kkambatl Karthik Kambatla (Inactive) added a comment - The patch doesn't apply to trunk anymore. Can someone update the patch? Thanks.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        updated the patch with only the new unit test, since it seems this bug is fixed already in trunk.
        The unit test succeeded.
        I suggest to include this unit test to have regression for this bug.

        Show
        zhiguohong Hong Zhiguo added a comment - updated the patch with only the new unit test, since it seems this bug is fixed already in trunk. The unit test succeeded. I suggest to include this unit test to have regression for this bug.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12666133/YARN-2306-2.patch
        against trunk revision 08a9ac7.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 3 warning messages.
        See https://builds.apache.org/job/PreCommit-YARN-Build/4806//artifact/trunk/patchprocess/diffJavadocWarnings.txt for details.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
        org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens

        The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TeTests
        org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservinTests
        org.apache.hadoop.yarn.server.resourcemanager.TesTests

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4806//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4806//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666133/YARN-2306-2.patch against trunk revision 08a9ac7. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. See https://builds.apache.org/job/PreCommit-YARN-Build/4806//artifact/trunk/patchprocess/diffJavadocWarnings.txt for details. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TeTests org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservinTests org.apache.hadoop.yarn.server.resourcemanager.TesTests +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4806//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4806//console This message is automatically generated.
        Hide
        ozawa Tsuyoshi Ozawa added a comment -

        Thanks for your updating, Hong Zhiguo.
        The test cases you added looks good to me. I know it's not related to your patch directory and minor nits, but how about stopping resourceManager in tearDown method to fix the test failures like this?

          @After
          public void tearDown() {
            scheduler = null;
            if (resourceManager != null) {
              resourceManager.stop();
            }
            QueueMetrics.clearQueueMetrics();
            DefaultMetricsSystem.shutdown();
          }
        
        Show
        ozawa Tsuyoshi Ozawa added a comment - Thanks for your updating, Hong Zhiguo . The test cases you added looks good to me. I know it's not related to your patch directory and minor nits, but how about stopping resourceManager in tearDown method to fix the test failures like this? @After public void tearDown() { scheduler = null ; if (resourceManager != null ) { resourceManager.stop(); } QueueMetrics.clearQueueMetrics(); DefaultMetricsSystem.shutdown(); }
        Hide
        ozawa Tsuyoshi Ozawa added a comment -

        s/directory/directly/

        Show
        ozawa Tsuyoshi Ozawa added a comment - s/directory/directly/
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12666133/YARN-2306-2.patch
        against trunk revision fe5c23b.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7053//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7053//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666133/YARN-2306-2.patch against trunk revision fe5c23b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7053//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7053//console This message is automatically generated.
        Hide
        john.jian.fang Jian Fang added a comment -

        Could someone please tell me which JIRA has fixed this bug in trunk? I am working on hadoop 2.6.0 branch and need to see if I need to fix this issue or not. Thanks in advance.

        Show
        john.jian.fang Jian Fang added a comment - Could someone please tell me which JIRA has fixed this bug in trunk? I am working on hadoop 2.6.0 branch and need to see if I need to fix this issue or not. Thanks in advance.
        Hide
        rchiang Ray Chiang added a comment -

        Running against trunk, I got failures 3 times in 3 runs.

        Show
        rchiang Ray Chiang added a comment - Running against trunk, I got failures 3 times in 3 runs.
        Hide
        kasha Karthik Kambatla added a comment -

        Canceling patch until review comments are addressed.

        Show
        kasha Karthik Kambatla added a comment - Canceling patch until review comments are addressed.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        hi, Ray Chiang, do you mean running the unit test in patch againt trunk?

        Show
        zhiguohong Hong Zhiguo added a comment - hi, Ray Chiang , do you mean running the unit test in patch againt trunk?
        Hide
        rchiang Ray Chiang added a comment -

        Heh. That was two months ago. I believe I was referring to the unit test.

        Show
        rchiang Ray Chiang added a comment - Heh. That was two months ago. I believe I was referring to the unit test.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        updated the patch againt trunk.

        Show
        zhiguohong Hong Zhiguo added a comment - updated the patch againt trunk.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        I checked the code of tearDown and it shows someone already did this.

        Show
        zhiguohong Hong Zhiguo added a comment - I checked the code of tearDown and it shows someone already did this.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Updated the patch. I ran testReservationMetrics several times and no failure now.

        Show
        zhiguohong Hong Zhiguo added a comment - Updated the patch. I ran testReservationMetrics several times and no failure now.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 9m 57s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 10m 1s There were no new javac warning messages.
        +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 2s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 39s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 52m 5s Tests failed in hadoop-yarn-server-resourcemanager.
            77m 36s  



        Reason Tests
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12745751/YARN-2306-3.patch
        Optional Tests javac unit findbugs checkstyle
        git revision trunk / ee36f4f
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8567/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8567/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8567/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 9m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 10m 1s There were no new javac warning messages. +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 2s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 52m 5s Tests failed in hadoop-yarn-server-resourcemanager.     77m 36s   Reason Tests Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745751/YARN-2306-3.patch Optional Tests javac unit findbugs checkstyle git revision trunk / ee36f4f hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8567/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8567/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8567/console This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        Hi Hong Zhiguo, are you still working on it? If not, can I pick it up?

        Show
        yufeigu Yufei Gu added a comment - Hi Hong Zhiguo , are you still working on it? If not, can I pick it up?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 41s trunk passed
        +1 compile 0m 35s trunk passed with JDK v1.8.0_77
        +1 compile 0m 30s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 1m 10s trunk passed
        +1 javadoc 0m 22s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.8.0_77
        +1 javac 0m 26s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.7.0_95
        +1 javac 0m 28s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 20s the patch passed
        +1 javadoc 0m 22s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_95
        -1 unit 68m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77.
        -1 unit 49m 55s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        136m 3s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.yarn.webapp.TestRMWithCSRFFilter
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
        JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
          hadoop.yarn.webapp.TestRMWithCSRFFilter
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12745751/YARN-2306-3.patch
        JIRA Issue YARN-2306
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bf8192cccc43 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0d1c115
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11078/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11078/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11078/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/11078/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11078/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11078/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 41s trunk passed +1 compile 0m 35s trunk passed with JDK v1.8.0_77 +1 compile 0m 30s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 10s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 31s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_77 +1 javac 0m 26s the patch passed +1 compile 0m 28s the patch passed with JDK v1.7.0_95 +1 javac 0m 28s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 20s the patch passed +1 javadoc 0m 22s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_95 -1 unit 68m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77. -1 unit 49m 55s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 136m 3s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps   hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12745751/YARN-2306-3.patch JIRA Issue YARN-2306 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bf8192cccc43 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0d1c115 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11078/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11078/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11078/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/11078/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11078/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11078/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        The patch is available, do you have any comments?

        Show
        zhiguohong Hong Zhiguo added a comment - The patch is available, do you have any comments?
        Hide
        yufeigu Yufei Gu added a comment -

        Hi Hong Zhiguo, Thank you for working on this. I am glad you are still working on it. Have you address all comments and the test failures from Hadoop QA?

        Show
        yufeigu Yufei Gu added a comment - Hi Hong Zhiguo , Thank you for working on this. I am glad you are still working on it. Have you address all comments and the test failures from Hadoop QA?
        Hide
        yufeigu Yufei Gu added a comment -

        YARN-2399 fixed this bug in trunk.

        Show
        yufeigu Yufei Gu added a comment - YARN-2399 fixed this bug in trunk.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 45s trunk passed
        +1 compile 0m 38s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 44s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 7s trunk passed
        +1 javadoc 0m 24s trunk passed
        +1 mvninstall 0m 38s the patch passed
        +1 compile 0m 36s the patch passed
        +1 javac 0m 36s the patch passed
        -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 205 unchanged - 0 fixed = 207 total (was 205)
        +1 mvnsite 0m 41s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 0m 21s the patch passed
        +1 unit 35m 39s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        52m 49s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-2306
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12745751/YARN-2306-3.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 24259f9c2a3e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b4a8fbc
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13576/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13576/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13576/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 45s trunk passed +1 compile 0m 38s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 44s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 7s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 0m 36s the patch passed +1 javac 0m 36s the patch passed -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 205 unchanged - 0 fixed = 207 total (was 205) +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 21s the patch passed +1 unit 35m 39s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 52m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-2306 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12745751/YARN-2306-3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 24259f9c2a3e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b4a8fbc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13576/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13576/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13576/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        The patch v3 is not applied to trunk. I uploaded the v4.

        Show
        yufeigu Yufei Gu added a comment - The patch v3 is not applied to trunk. I uploaded the v4.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 44s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 0m 43s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 5s the patch passed
        +1 javadoc 0m 21s the patch passed
        -1 unit 36m 39s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        53m 13s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMHA



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-2306
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835675/YARN-2306.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8e3b32dcd3da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 79ae78d
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13590/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13590/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13590/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 44s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 43s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 21s the patch passed -1 unit 36m 39s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 53m 13s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-2306 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835675/YARN-2306.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8e3b32dcd3da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 79ae78d Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/13590/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13590/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13590/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        The test failure is unrelated.

        Show
        yufeigu Yufei Gu added a comment - The test failure is unrelated.
        Hide
        subru Subru Krishnan added a comment -

        Thanks Hong Zhiguo and Yufei Gu for the patch, I committed it to trunk/branch-2.

        Show
        subru Subru Krishnan added a comment - Thanks Hong Zhiguo and Yufei Gu for the patch, I committed it to trunk/branch-2.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10715 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10715/)
        YARN-2306. Add test for leakage of reservation metrics in fair (subu: rev b2c4f24c31e73faa8f71d44db5de3aa91e3b7d5e)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10715 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10715/ ) YARN-2306 . Add test for leakage of reservation metrics in fair (subu: rev b2c4f24c31e73faa8f71d44db5de3aa91e3b7d5e) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java

          People

          • Assignee:
            zhiguohong Hong Zhiguo
            Reporter:
            zhiguohong Hong Zhiguo
          • Votes:
            1 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development