Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4418

AM Resource Limit per partition can be updated to ResourceUsage as well

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: resourcemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      AMResourceLimit is now extended to all partitions after YARN-3216. Its also better to track this ResourceLimit in existing ResourceUsage so that REST framework can be benefited to avail this information easily.

      1. 0001-YARN-4418.patch
        10 kB
        Sunil G
      2. 0002-YARN-4418.patch
        6 kB
        Sunil G
      3. 0003-YARN-4418.patch
        6 kB
        Sunil G
      4. 0004-YARN-4418.patch
        4 kB
        Sunil G
      5. 0005-YARN-4418.patch
        9 kB
        Sunil G

        Activity

        Hide
        sunilg Sunil G added a comment -

        Attaching an initial patch for updating AM Limit per partition limit to ResourceUsage for queue.

        I was trying to update latest {{AM Limit per partition }} just before creating DAO object. This can help in getting latest and updated limit info (for NO_LABEL, we may consider queue used capacity which can go upto max-capacity and limit will vary in such cases).
        Wangda Tanpls help to review the same and share your thoughts.

        Show
        sunilg Sunil G added a comment - Attaching an initial patch for updating AM Limit per partition limit to ResourceUsage for queue. I was trying to update latest {{AM Limit per partition }} just before creating DAO object. This can help in getting latest and updated limit info (for NO_LABEL, we may consider queue used capacity which can go upto max-capacity and limit will vary in such cases). Wangda Tan pls help to review the same and share your thoughts.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 54s trunk passed
        +1 compile 0m 41s trunk passed with JDK v1.8.0_66
        +1 compile 0m 39s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 47s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 1m 34s trunk passed
        +1 javadoc 0m 33s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 35s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 44s the patch passed with JDK v1.8.0_66
        +1 javac 0m 44s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_85
        +1 javac 0m 41s the patch passed
        -1 checkstyle 0m 16s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 161, now 163).
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 18s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 33s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 36s the patch passed with JDK v1.7.0_85
        -1 unit 64m 58s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 67m 42s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85.
        -1 asflicense 0m 20s Patch generated 1 ASF License warnings.
        156m 17s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
        JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775781/0001-YARN-4418.patch
        JIRA Issue YARN-4418
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2e26f0d65b7e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 86c95cb
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9871/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9871/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 54s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_66 +1 compile 0m 39s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 47s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 34s trunk passed +1 javadoc 0m 33s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 35s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 46s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_66 +1 javac 0m 44s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_85 +1 javac 0m 41s the patch passed -1 checkstyle 0m 16s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 161, now 163). +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 33s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 36s the patch passed with JDK v1.7.0_85 -1 unit 64m 58s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 67m 42s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 20s Patch generated 1 ASF License warnings. 156m 17s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions JDK v1.7.0_85 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775781/0001-YARN-4418.patch JIRA Issue YARN-4418 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2e26f0d65b7e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 86c95cb findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9871/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9871/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9871/console This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Uploading a newer version of patching after looking into test failures.

        One point to note:
        AMLimit is now a part of ResourceUsage. ResourceUsage was updated when containers were allocated to it, else there wont be any entries added to the ResourceUsage. However, AMLimit may slightly vary here as it need not have to wait for any real resourceUsage to get it calculated. It can take queue/partitionCapacity to get the capacity.

        In first patch, I tried to calculate/update AMLimit for all accessible labels. I was slightly in half mind here, due to some unwanted resource usage updates here came also with this (0 for all other enums in ResourceUsage) . So I changed to update AMLimit for those labels which are now Used/Added in ResourceUsage.

        Wangda Tan pls help to share your point of view here so that we can finalize on this.

        Show
        sunilg Sunil G added a comment - Uploading a newer version of patching after looking into test failures. One point to note: AMLimit is now a part of ResourceUsage. ResourceUsage was updated when containers were allocated to it, else there wont be any entries added to the ResourceUsage. However, AMLimit may slightly vary here as it need not have to wait for any real resourceUsage to get it calculated. It can take queue/partitionCapacity to get the capacity. In first patch, I tried to calculate/update AMLimit for all accessible labels. I was slightly in half mind here, due to some unwanted resource usage updates here came also with this (0 for all other enums in ResourceUsage) . So I changed to update AMLimit for those labels which are now Used/Added in ResourceUsage. Wangda Tan pls help to share your point of view here so that we can finalize on this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 41s trunk passed
        +1 compile 0m 34s trunk passed with JDK v1.8.0_66
        +1 compile 0m 29s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 9s trunk passed
        -1 javadoc 0m 20s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66.
        +1 javadoc 0m 26s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.8.0_66
        +1 javac 0m 26s the patch passed
        +1 compile 0m 30s the patch passed with JDK v1.7.0_91
        +1 javac 0m 30s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 18s the patch passed
        -1 javadoc 0m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91
        -1 unit 63m 11s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 64m 52s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        145m 49s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776124/0002-YARN-4418.patch
        JIRA Issue YARN-4418
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 53fc4fb273c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 01a641b
        findbugs v3.0.0
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9886/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9886/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 41s trunk passed +1 compile 0m 34s trunk passed with JDK v1.8.0_66 +1 compile 0m 29s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 9s trunk passed -1 javadoc 0m 20s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66. +1 javadoc 0m 26s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 33s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_66 +1 javac 0m 26s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_91 +1 javac 0m 30s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed -1 javadoc 0m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91 -1 unit 63m 11s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 64m 52s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 145m 49s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776124/0002-YARN-4418.patch JIRA Issue YARN-4418 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 53fc4fb273c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 01a641b findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9886/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9886/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9886/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sunil G,

        I'm not sure if we need to update am limit when called by REST API.

        • getAMResourceLimitPerPartition has set AMLimit
        • activateApplications will be called when doing updateClusterResource
        • updateClusterResource will be called when resource of the cluster updated OR queue reinitialized
        • AMResourceLimit will be changed only when resource the the cluster updated OR queue reinitialized

        So I think getAMResourceLimit should always up-to-date and changes of CapacitySchedulerLeafQueueInfo should be reverted.

        Thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Sunil G , I'm not sure if we need to update am limit when called by REST API. getAMResourceLimitPerPartition has set AMLimit activateApplications will be called when doing updateClusterResource updateClusterResource will be called when resource of the cluster updated OR queue reinitialized AMResourceLimit will be changed only when resource the the cluster updated OR queue reinitialized So I think getAMResourceLimit should always up-to-date and changes of CapacitySchedulerLeafQueueInfo should be reverted. Thoughts?
        Hide
        sunilg Sunil G added a comment -

        Thank you Wangda Tan for clarifying the scenario.
        I still have one doubt here, getAMResourceLimitPerPartition will try to get the Queue's current usage (it can use more than allocated capacity) in case when no labels are configured. It can happen in case where some more containers got allocated rather than some new application. In this case getAMResourceLimitPerPartition wont be invoked and if we calcuate the AMLimit at this time, it will be lil higher (since queue is running top of its allocated capcity, going towards max-capcity).

        This was the only corner case which I felt, other than that all cases are handled as you mentioned. So I could try only to get NO_LABEL partition from DAO side? Do you see this as a valid case. pls share your thoughts.

        Show
        sunilg Sunil G added a comment - Thank you Wangda Tan for clarifying the scenario. I still have one doubt here, getAMResourceLimitPerPartition will try to get the Queue's current usage (it can use more than allocated capacity) in case when no labels are configured. It can happen in case where some more containers got allocated rather than some new application. In this case getAMResourceLimitPerPartition wont be invoked and if we calcuate the AMLimit at this time, it will be lil higher (since queue is running top of its allocated capcity, going towards max-capcity). This was the only corner case which I felt, other than that all cases are handled as you mentioned. So I could try only to get NO_LABEL partition from DAO side? Do you see this as a valid case. pls share your thoughts.
        Hide
        sunilg Sunil G added a comment -

        Uploading a patch by handling only NO_LABEL scenario. Wangda Tan pls help to check this corner case, and if its not valid I remove the check.

        Show
        sunilg Sunil G added a comment - Uploading a patch by handling only NO_LABEL scenario. Wangda Tan pls help to check this corner case, and if its not valid I remove the check.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 1s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 23s trunk passed
        +1 compile 0m 35s trunk passed with JDK v1.8.0_66
        +1 compile 0m 34s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 41s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 1m 20s trunk passed
        -1 javadoc 0m 26s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66.
        +1 javadoc 0m 30s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 38s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.8.0_66
        +1 javac 0m 38s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.7.0_91
        +1 javac 0m 37s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 43s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 31s the patch passed
        -1 javadoc 0m 26s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 javadoc 0m 30s the patch passed with JDK v1.7.0_91
        -1 unit 66m 3s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 66m 23s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 27s Patch does not generate ASF License warnings.
        168m 11s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMService
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776602/0003-YARN-4418.patch
        JIRA Issue YARN-4418
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c3fb572a61bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 50edcb9
        findbugs v3.0.0
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9913/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9913/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 23s trunk passed +1 compile 0m 35s trunk passed with JDK v1.8.0_66 +1 compile 0m 34s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 41s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 20s trunk passed -1 javadoc 0m 26s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66. +1 javadoc 0m 30s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 38s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_66 +1 javac 0m 38s the patch passed +1 compile 0m 37s the patch passed with JDK v1.7.0_91 +1 javac 0m 37s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 43s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 31s the patch passed -1 javadoc 0m 26s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 javadoc 0m 30s the patch passed with JDK v1.7.0_91 -1 unit 66m 3s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 66m 23s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 168m 11s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMService JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776602/0003-YARN-4418.patch JIRA Issue YARN-4418 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c3fb572a61bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 50edcb9 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9913/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9913/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9913/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sunil G,

        Thanks for explaining, I feel we shouldn't calculate am-resource-limit based on available resource.

        It is possible that a queue's available resource is just temporarily more than queue's configured capacity. It may over allocate AM container. Do you agree?

        Show
        leftnoteasy Wangda Tan added a comment - Sunil G , Thanks for explaining, I feel we shouldn't calculate am-resource-limit based on available resource. It is possible that a queue's available resource is just temporarily more than queue's configured capacity. It may over allocate AM container. Do you agree?
        Hide
        sunilg Sunil G added a comment -

        Thank you Wangda Tan for the explanation. That sounds good.
        So as you mentioned earlier, we will have this metric updated only when activateApplications is called. and this covers all cases. Uploading a patch addressing same. Kindly help to check.

        Show
        sunilg Sunil G added a comment - Thank you Wangda Tan for the explanation. That sounds good. So as you mentioned earlier, we will have this metric updated only when activateApplications is called. and this covers all cases. Uploading a patch addressing same. Kindly help to check.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 10m 24s trunk passed
        +1 compile 0m 45s trunk passed with JDK v1.8.0_66
        +1 compile 0m 40s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 48s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 1m 35s trunk passed
        -1 javadoc 0m 34s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66.
        +1 javadoc 0m 37s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 43s the patch passed with JDK v1.8.0_66
        +1 javac 0m 43s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_91
        +1 javac 0m 41s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        -1 javadoc 0m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 javadoc 0m 36s the patch passed with JDK v1.7.0_91
        -1 unit 66m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 67m 2s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 32s Patch does not generate ASF License warnings.
        158m 26s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776705/0004-YARN-4418.patch
        JIRA Issue YARN-4418
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux dd323d6682e0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 132478e
        findbugs v3.0.0
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9921/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9921/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 24s trunk passed +1 compile 0m 45s trunk passed with JDK v1.8.0_66 +1 compile 0m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 48s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 35s trunk passed -1 javadoc 0m 34s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66. +1 javadoc 0m 37s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 46s the patch passed +1 compile 0m 43s the patch passed with JDK v1.8.0_66 +1 javac 0m 43s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_91 +1 javac 0m 41s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 45s the patch passed -1 javadoc 0m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 javadoc 0m 36s the patch passed with JDK v1.7.0_91 -1 unit 66m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 67m 2s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 32s Patch does not generate ASF License warnings. 158m 26s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776705/0004-YARN-4418.patch JIRA Issue YARN-4418 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dd323d6682e0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 132478e findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9921/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9921/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9921/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Hi Sunil G,
        Patch generally looks good, thanks for update. Could you add AMLimit, CachedPending and CachedUsed to TestResourceUsage as well?

        Show
        leftnoteasy Wangda Tan added a comment - Hi Sunil G , Patch generally looks good, thanks for update. Could you add AMLimit, CachedPending and CachedUsed to TestResourceUsage as well?
        Hide
        leftnoteasy Wangda Tan added a comment -

        And also, TestQueueCapacities should be synced to QueueCapacities as well. Could you update them together in this patch?

        Show
        leftnoteasy Wangda Tan added a comment - And also, TestQueueCapacities should be synced to QueueCapacities as well. Could you update them together in this patch?
        Hide
        sunilg Sunil G added a comment -

        Thank you Wangda Tan for the comments.
        Updated patch by adding parameterized test cases for missing entries in ResourceUsage and QueueCapacities in to their respective test classes. Kindly help to review.

        Show
        sunilg Sunil G added a comment - Thank you Wangda Tan for the comments. Updated patch by adding parameterized test cases for missing entries in ResourceUsage and QueueCapacities in to their respective test classes. Kindly help to review.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 8m 4s trunk passed
        +1 compile 0m 30s trunk passed with JDK v1.8.0_66
        +1 compile 0m 32s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 39s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 17s trunk passed
        -1 javadoc 0m 25s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66.
        +1 javadoc 0m 29s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 37s the patch passed
        +1 compile 0m 29s the patch passed with JDK v1.8.0_66
        +1 javac 0m 29s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.7.0_91
        +1 javac 0m 34s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 41s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 25s the patch passed
        -1 javadoc 0m 23s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 javadoc 0m 29s the patch passed with JDK v1.7.0_91
        -1 unit 65m 43s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 66m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        150m 51s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
          hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777272/0005-YARN-4418.patch
        JIRA Issue YARN-4418
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8ba62f348112 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bf5295b
        findbugs v3.0.0
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9947/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 75MB
        Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9947/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 4s trunk passed +1 compile 0m 30s trunk passed with JDK v1.8.0_66 +1 compile 0m 32s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 17s trunk passed -1 javadoc 0m 25s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66. +1 javadoc 0m 29s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 37s the patch passed +1 compile 0m 29s the patch passed with JDK v1.8.0_66 +1 javac 0m 29s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_91 +1 javac 0m 34s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 25s the patch passed -1 javadoc 0m 23s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 javadoc 0m 29s the patch passed with JDK v1.7.0_91 -1 unit 65m 43s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 66m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 150m 51s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens   hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777272/0005-YARN-4418.patch JIRA Issue YARN-4418 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8ba62f348112 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf5295b findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9947/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9947/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9947/console This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Apart from known test case failures, others are passing locally.

        Show
        sunilg Sunil G added a comment - Apart from known test case failures, others are passing locally.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Looks good, thanks Sunil G. Committing..

        Show
        leftnoteasy Wangda Tan added a comment - Looks good, thanks Sunil G . Committing..
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to trunk/branch-2, thanks Sunil G!

        Show
        leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2, thanks Sunil G !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8963 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8963/)
        YARN-4418. AM Resource Limit per partition can be updated to (wangda: rev 07b0fb996a32020678bd2ce482b672f0434651f0)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestResourceUsage.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8963 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8963/ ) YARN-4418 . AM Resource Limit per partition can be updated to (wangda: rev 07b0fb996a32020678bd2ce482b672f0434651f0) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestResourceUsage.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #692 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/692/)
        YARN-4418. AM Resource Limit per partition can be updated to (wangda: rev 07b0fb996a32020678bd2ce482b672f0434651f0)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestResourceUsage.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #692 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/692/ ) YARN-4418 . AM Resource Limit per partition can be updated to (wangda: rev 07b0fb996a32020678bd2ce482b672f0434651f0) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueCapacities.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/QueueCapacities.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestResourceUsage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java
        Hide
        sunilg Sunil G added a comment -

        Thanks you very much Wangda Tan for the review and commit.

        Show
        sunilg Sunil G added a comment - Thanks you very much Wangda Tan for the review and commit.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to branch-2.8.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to branch-2.8.

          People

          • Assignee:
            sunilg Sunil G
            Reporter:
            sunilg Sunil G
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development