Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4878

Expose scheduling policy and max running apps over JMX for Yarn queues

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: yarn
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There are two things that are not currently visible over JMX: the current scheduling policy for a queue, and the number of max running apps. It would be great if these could be exposed over JMX as well.

      1. YARN-4878.001.patch
        7 kB
        Yufei Gu
      2. YARN-4878.002.patch
        7 kB
        Yufei Gu
      3. YARN-4878.003.patch
        7 kB
        Yufei Gu
      4. YARN-4878.004.patch
        7 kB
        Yufei Gu

        Activity

        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Karthik Kambatla for the review and commit.
        Thanks Ray Chiang for the review.

        Show
        yufeigu Yufei Gu added a comment - Thanks Karthik Kambatla for the review and commit. Thanks Ray Chiang for the review.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9848 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9848/)
        YARN-4878. Expose scheduling policy and max running apps over JMX for (kasha: rev f979d779e192ac05906b7eb880dfc8884d0078a0)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9848 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9848/ ) YARN-4878 . Expose scheduling policy and max running apps over JMX for (kasha: rev f979d779e192ac05906b7eb880dfc8884d0078a0) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java
        Hide
        kasha Karthik Kambatla added a comment -

        Thanks for the contribution, Yufei. Just committed this to trunk and branch-2.

        Show
        kasha Karthik Kambatla added a comment - Thanks for the contribution, Yufei. Just committed this to trunk and branch-2.
        Hide
        kasha Karthik Kambatla added a comment -

        +1, checking this in.

        Show
        kasha Karthik Kambatla added a comment - +1, checking this in.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 24s trunk passed
        +1 compile 0m 28s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 56s trunk passed
        +1 javadoc 0m 24s trunk passed
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 34s the patch passed
        +1 javac 0m 34s the patch passed
        -1 checkstyle 0m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 52 unchanged - 0 fixed = 53 total (was 52)
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 58s the patch passed
        +1 javadoc 0m 19s the patch passed
        -1 unit 31m 25s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s Patch does not generate ASF License warnings.
        46m 17s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805051/YARN-4878.004.patch
        JIRA Issue YARN-4878
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 05d5f14751de 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 22fcd81
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11570/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11570/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11570/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11570/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11570/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 24s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed -1 checkstyle 0m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 52 unchanged - 0 fixed = 53 total (was 52) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 58s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 31m 25s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 46m 17s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805051/YARN-4878.004.patch JIRA Issue YARN-4878 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 05d5f14751de 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22fcd81 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11570/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11570/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11570/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11570/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11570/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        The patch 004 fixed the style issue except the first one. The first style issue should not be fixed.

        Show
        yufeigu Yufei Gu added a comment - The patch 004 fixed the style issue except the first one. The first style issue should not be fixed.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 10s trunk passed
        +1 compile 0m 28s trunk passed
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 34s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 56s trunk passed
        +1 javadoc 0m 23s trunk passed
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 25s the patch passed
        +1 javac 0m 25s the patch passed
        -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 4 new + 53 unchanged - 0 fixed = 57 total (was 53)
        +1 mvnsite 0m 30s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 3s the patch passed
        +1 javadoc 0m 20s the patch passed
        -1 unit 31m 10s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 15s Patch does not generate ASF License warnings.
        44m 29s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804817/YARN-4878.003.patch
        JIRA Issue YARN-4878
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 22763cbdee58 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1597630
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11546/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11546/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11546/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11546/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11546/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 10s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 4 new + 53 unchanged - 0 fixed = 57 total (was 53) +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 31m 10s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 44m 29s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804817/YARN-4878.003.patch JIRA Issue YARN-4878 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 22763cbdee58 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11546/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11546/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11546/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11546/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11546/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 16s trunk passed
        +1 compile 0m 28s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 51s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 26s the patch passed
        +1 javac 0m 26s the patch passed
        -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 4 new + 53 unchanged - 0 fixed = 57 total (was 53)
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 58s the patch passed
        +1 javadoc 0m 18s the patch passed
        -1 unit 31m 2s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        44m 21s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804817/YARN-4878.003.patch
        JIRA Issue YARN-4878
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a97e72ccbd0d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 010e6ac
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11537/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11537/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11537/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11537/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11537/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 16s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 51s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 4 new + 53 unchanged - 0 fixed = 57 total (was 53) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 58s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 31m 2s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 44m 21s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804817/YARN-4878.003.patch JIRA Issue YARN-4878 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a97e72ccbd0d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 010e6ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11537/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11537/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11537/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11537/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11537/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        My visual diff has the same problem. I tried some ways to get ride of it, but failed. Thanks Ray Chiang.

        Show
        yufeigu Yufei Gu added a comment - My visual diff has the same problem. I tried some ways to get ride of it, but failed. Thanks Ray Chiang .
        Hide
        rchiang Ray Chiang added a comment -

        Oh weird. My visual diff made it look like "public synchronized" was added by you. Looks fine to leave alone.

        Show
        rchiang Ray Chiang added a comment - Oh weird. My visual diff made it look like "public synchronized" was added by you. Looks fine to leave alone.
        Hide
        yufeigu Yufei Gu added a comment -

        Upload patch 003 for Ray Chiang's comments.

        Show
        yufeigu Yufei Gu added a comment - Upload patch 003 for Ray Chiang 's comments.
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Ray Chiang for reviewing.

        1. Nice point.
        2. Since function forqueue is unrelated to this JIRA, should we just leave it?

        Show
        yufeigu Yufei Gu added a comment - Thanks Ray Chiang for reviewing. 1. Nice point. 2. Since function forqueue is unrelated to this JIRA, should we just leave it?
        Hide
        rchiang Ray Chiang added a comment -

        Some minor nitpicks:

        • Change the metric description from "Maximum applications number" to "Maximum number of applications"
        • For readability, I'd recommend changing this style:
          public synchronized
          static FSQueueMetrics forQueue(String queueName, Queue parent,
              boolean enableUserMetrics, Configuration conf) {
        

        to this:

          public static synchronized FSQueueMetrics forQueue(String queueName,
              Queue parent, boolean enableUserMetrics, Configuration conf) {
        

        In general, it's best to have the code readable, not best diff compatibility.

        Show
        rchiang Ray Chiang added a comment - Some minor nitpicks: Change the metric description from "Maximum applications number" to "Maximum number of applications" For readability, I'd recommend changing this style: public synchronized static FSQueueMetrics forQueue(String queueName, Queue parent, boolean enableUserMetrics, Configuration conf) { to this: public static synchronized FSQueueMetrics forQueue(String queueName, Queue parent, boolean enableUserMetrics, Configuration conf) { In general, it's best to have the code readable, not best diff compatibility.
        Hide
        yufeigu Yufei Gu added a comment -

        I've tested all failed test cases. All of them are unrelated to the patch.
        In terms of style error, I mimic the existing code, I think it's better to follow the convention of the existing code in the class.

        Show
        yufeigu Yufei Gu added a comment - I've tested all failed test cases. All of them are unrelated to the patch. In terms of style error, I mimic the existing code, I think it's better to follow the convention of the existing code in the class.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 55s trunk passed
        +1 compile 0m 28s trunk passed with JDK v1.8.0_77
        +1 compile 0m 30s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 1m 5s trunk passed
        +1 javadoc 0m 22s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.8.0_77
        +1 javac 0m 24s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.7.0_95
        +1 javac 0m 26s the patch passed
        -1 checkstyle 0m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 38 unchanged - 0 fixed = 39 total (was 38)
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 16s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 23s the patch passed with JDK v1.7.0_95
        -1 unit 78m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77.
        -1 unit 57m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        153m 10s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.webapp.TestRMWithCSRFFilter
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
        JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.webapp.TestRMWithCSRFFilter
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798390/YARN-4878.002.patch
        JIRA Issue YARN-4878
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bc7de22403a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 042a3ae
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11056/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11056/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 55s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_77 +1 compile 0m 30s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 31s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_77 +1 javac 0m 24s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed -1 checkstyle 0m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 38 unchanged - 0 fixed = 39 total (was 38) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_95 -1 unit 78m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77. -1 unit 57m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 153m 10s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798390/YARN-4878.002.patch JIRA Issue YARN-4878 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bc7de22403a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 042a3ae Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/11056/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11056/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11056/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        Hi Karthik Kambatla, thanks for the review.
        1. I tried scheduler.allocConf. But lots of test cases failed. I figured out and fixed them in the second patch.
        2. I did this to mimic the existing code. If we are going to consolidate both of them, can we do it in a followup JIRA?

        Show
        yufeigu Yufei Gu added a comment - Hi Karthik Kambatla , thanks for the review. 1. I tried scheduler.allocConf . But lots of test cases failed. I figured out and fixed them in the second patch. 2. I did this to mimic the existing code. If we are going to consolidate both of them, can we do it in a followup JIRA?
        Hide
        kasha Karthik Kambatla added a comment -

        Thanks for adding these, Yufei. Comments on the patch:

        1. FSQueue constructor: instead of creating a new AllocationConfiguration, use scheduler.allocConf. You probably don't need another local variable either.
        2. Why do we need to set the metrics in both QueueManager and FSQueue?
        Show
        kasha Karthik Kambatla added a comment - Thanks for adding these, Yufei. Comments on the patch: FSQueue constructor: instead of creating a new AllocationConfiguration, use scheduler.allocConf . You probably don't need another local variable either. Why do we need to set the metrics in both QueueManager and FSQueue?
        Hide
        yufeigu Yufei Gu added a comment -

        Hi Karthik Kambatla , I just uploaded the patch. Would you please review it? Thanks.

        Show
        yufeigu Yufei Gu added a comment - Hi Karthik Kambatla , I just uploaded the patch. Would you please review it? Thanks.

          People

          • Assignee:
            yufeigu Yufei Gu
            Reporter:
            yufeigu Yufei Gu
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development