Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7738

CapacityScheduler: Support refresh maximum allocation for multiple resource types

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 3.1.0
    • None
    • None
    • Reviewed

    Description

      Currently CapacityScheduler fails to refresh maximum allocation for multiple resource types.

      Attachments

        1. YARN-7738.001.patch
          13 kB
          Wangda Tan
        2. YARN-7738.002.patch
          23 kB
          Wangda Tan
        3. YARN-7738.003.patch
          24 kB
          Sunil G
        4. YARN-7738.004.patch
          24 kB
          Sunil G

        Issue Links

          Activity

            leftnoteasy Wangda Tan added a comment -

            Thanks ssathish@hortonworks.com for reporting this issue, updated title/desc and will put a patch soon.

            leftnoteasy Wangda Tan added a comment - Thanks ssathish@hortonworks.com for reporting this issue, updated title/desc and will put a patch soon.
            leftnoteasy Wangda Tan added a comment -

            cc: daniel@cloudera.com, I'm not sure if FS supports refresh maximum allocation while RM is running, if yes, FS needs this fix as well.

            leftnoteasy Wangda Tan added a comment - cc: daniel@cloudera.com , I'm not sure if FS supports refresh maximum allocation while RM is running, if yes, FS needs this fix as well.
            leftnoteasy Wangda Tan added a comment -

            Attached ver.1 patch and verified it works on a cluster with GPU enabled.

            Will add test cases in the next update.

            leftnoteasy Wangda Tan added a comment - Attached ver.1 patch and verified it works on a cluster with GPU enabled. Will add test cases in the next update.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 9m 41s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 59s Maven dependency ordering for branch
            +1 mvninstall 18m 1s trunk passed
            +1 compile 8m 7s trunk passed
            +1 checkstyle 1m 6s trunk passed
            +1 mvnsite 1m 26s trunk passed
            +1 shadedclient 12m 42s branch has no errors when building and testing our client artifacts.
            -1 findbugs 1m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 0m 58s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 10s Maven dependency ordering for patch
            +1 mvninstall 1m 9s the patch passed
            +1 compile 6m 31s the patch passed
            +1 javac 6m 31s the patch passed
            -0 checkstyle 1m 16s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 367 unchanged - 0 fixed = 369 total (was 367)
            +1 mvnsite 1m 25s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 10m 56s patch has no errors when building and testing our client artifacts.
            +1 findbugs 2m 37s the patch passed
            +1 javadoc 1m 0s the patch passed
                  Other Tests
            +1 unit 0m 39s hadoop-yarn-api in the patch passed.
            -1 unit 59m 50s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 32s The patch does not generate ASF License warnings.
            140m 50s



            Reason Tests
            Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7738
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905765/YARN-7738.001.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 9106d45b396a 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / bc285da
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19208/testReport/
            Max. process+thread count 799 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/19208/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 9m 41s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 59s Maven dependency ordering for branch +1 mvninstall 18m 1s trunk passed +1 compile 8m 7s trunk passed +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 26s trunk passed +1 shadedclient 12m 42s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 58s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 6m 31s the patch passed +1 javac 6m 31s the patch passed -0 checkstyle 1m 16s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 367 unchanged - 0 fixed = 369 total (was 367) +1 mvnsite 1m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 10m 56s patch has no errors when building and testing our client artifacts. +1 findbugs 2m 37s the patch passed +1 javadoc 1m 0s the patch passed       Other Tests +1 unit 0m 39s hadoop-yarn-api in the patch passed. -1 unit 59m 50s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 140m 50s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905765/YARN-7738.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 9106d45b396a 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / bc285da maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19208/testReport/ Max. process+thread count 799 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/19208/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            leftnoteasy Wangda Tan added a comment -

            Updated ver.2 patch, fixed use cases.

            leftnoteasy Wangda Tan added a comment - Updated ver.2 patch, fixed use cases.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 15s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 10s Maven dependency ordering for branch
            +1 mvninstall 16m 45s trunk passed
            +1 compile 8m 7s trunk passed
            +1 checkstyle 1m 5s trunk passed
            +1 mvnsite 1m 26s trunk passed
            +1 shadedclient 12m 51s branch has no errors when building and testing our client artifacts.
            -1 findbugs 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 0s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 18s the patch passed
            +1 compile 7m 28s the patch passed
            +1 javac 7m 28s the patch passed
            -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 413 unchanged - 0 fixed = 416 total (was 413)
            +1 mvnsite 1m 22s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 10m 50s patch has no errors when building and testing our client artifacts.
            +1 findbugs 2m 44s the patch passed
            +1 javadoc 1m 0s the patch passed
                  Other Tests
            +1 unit 0m 42s hadoop-yarn-api in the patch passed.
            +1 unit 66m 50s hadoop-yarn-server-resourcemanager in the patch passed.
            +1 asflicense 0m 37s The patch does not generate ASF License warnings.
            137m 37s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7738
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905812/YARN-7738.002.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux dfe31b26722f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / addbcd8
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19215/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19215/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19215/testReport/
            Max. process+thread count 843 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/19215/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 16m 45s trunk passed +1 compile 8m 7s trunk passed +1 checkstyle 1m 5s trunk passed +1 mvnsite 1m 26s trunk passed +1 shadedclient 12m 51s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 0s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 7m 28s the patch passed +1 javac 7m 28s the patch passed -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 413 unchanged - 0 fixed = 416 total (was 413) +1 mvnsite 1m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 10m 50s patch has no errors when building and testing our client artifacts. +1 findbugs 2m 44s the patch passed +1 javadoc 1m 0s the patch passed       Other Tests +1 unit 0m 42s hadoop-yarn-api in the patch passed. +1 unit 66m 50s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 137m 37s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905812/YARN-7738.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux dfe31b26722f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / addbcd8 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19215/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19215/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19215/testReport/ Max. process+thread count 843 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/19215/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            sunilg Sunil G added a comment -

            Thanks leftnoteasy. Fix seems fine to me

            Few comments:

            1. Resource#setResourceValue will internally take care of checking whether resource name is memory or vcores and accordingly call setMemorySize or setVirtualCores. Hence we could avoid setMaximumAllocationByConfiguredResourceInformation in ResourceUtils.
            2. if (!Resources.fitsIn(oldMax, newMax)) could we make its like if (Resources.fitsIn(newMax, oldMax)). fitsIn checks smaller as first arg.
              TO check test cases, i ll try attach a patch with this changes.
            sunilg Sunil G added a comment - Thanks leftnoteasy . Fix seems fine to me Few comments: Resource#setResourceValue will internally take care of checking whether resource name is memory or vcores and accordingly call setMemorySize or setVirtualCores. Hence we could avoid setMaximumAllocationByConfiguredResourceInformation in ResourceUtils. if (!Resources.fitsIn(oldMax, newMax)) could we make its like if (Resources.fitsIn(newMax, oldMax)) . fitsIn checks smaller as first arg. TO check test cases, i ll try attach a patch with this changes.
            sunilg Sunil G added a comment -

            attaching patch for jenkins.

            sunilg Sunil G added a comment - attaching patch for jenkins.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 30s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 48s Maven dependency ordering for branch
            +1 mvninstall 15m 1s trunk passed
            +1 compile 9m 22s trunk passed
            +1 checkstyle 0m 57s trunk passed
            +1 mvnsite 1m 16s trunk passed
            +1 shadedclient 11m 1s branch has no errors when building and testing our client artifacts.
            -1 findbugs 1m 8s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 0s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 5s the patch passed
            +1 compile 8m 26s the patch passed
            +1 javac 8m 26s the patch passed
            -0 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415)
            +1 mvnsite 1m 13s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 9m 33s patch has no errors when building and testing our client artifacts.
            +1 findbugs 2m 26s the patch passed
            +1 javadoc 0m 48s the patch passed
                  Other Tests
            +1 unit 0m 34s hadoop-yarn-api in the patch passed.
            -1 unit 74m 32s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 33s The patch does not generate ASF License warnings.
            141m 43s



            Reason Tests
            Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
              hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
              hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing
              hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestZKConfigurationStore
              hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
              hadoop.yarn.server.resourcemanager.TestRMHAForAsyncScheduler
              hadoop.yarn.server.resourcemanager.TestLeaderElectorService
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation
              hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesConfigurationMutation
              hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState
              hadoop.yarn.server.resourcemanager.TestRMHA
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
              hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService
              hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations
              hadoop.yarn.server.resourcemanager.TestRMAdminService
              hadoop.yarn.server.resourcemanager.TestRMStoreCommands
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestAbsoluteResourceConfiguration
              hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
              hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestLeveldbConfigurationStore
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
              hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings
              hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
              hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7738
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905840/YARN-7738.003.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 1a13ac9a4554 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / addbcd8
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19217/testReport/
            Max. process+thread count 869 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/19217/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 15m 1s trunk passed +1 compile 9m 22s trunk passed +1 checkstyle 0m 57s trunk passed +1 mvnsite 1m 16s trunk passed +1 shadedclient 11m 1s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 8s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 0s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 5s the patch passed +1 compile 8m 26s the patch passed +1 javac 8m 26s the patch passed -0 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) +1 mvnsite 1m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 9m 33s patch has no errors when building and testing our client artifacts. +1 findbugs 2m 26s the patch passed +1 javadoc 0m 48s the patch passed       Other Tests +1 unit 0m 34s hadoop-yarn-api in the patch passed. -1 unit 74m 32s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 141m 43s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA   hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing   hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestZKConfigurationStore   hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors   hadoop.yarn.server.resourcemanager.TestRMHAForAsyncScheduler   hadoop.yarn.server.resourcemanager.TestLeaderElectorService   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesConfigurationMutation   hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState   hadoop.yarn.server.resourcemanager.TestRMHA   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService   hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestRMStoreCommands   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestAbsoluteResourceConfiguration   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestLeveldbConfigurationStore   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905840/YARN-7738.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 1a13ac9a4554 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / addbcd8 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19217/testReport/ Max. process+thread count 869 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/19217/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            sunilg Sunil G added a comment -

            fitsIn logic was originally correct. Reattaching the patch.

            sunilg Sunil G added a comment - fitsIn logic was originally correct. Reattaching the patch.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 docker 6m 59s Docker failed to build yetus/hadoop:5b98639.



            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 6m 59s Docker failed to build yetus/hadoop:5b98639. Subsystem Report/Notes JIRA Issue YARN-7738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/19221/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 docker 5m 38s Docker failed to build yetus/hadoop:5b98639.



            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 5m 38s Docker failed to build yetus/hadoop:5b98639. Subsystem Report/Notes JIRA Issue YARN-7738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/19222/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 18s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 10s Maven dependency ordering for branch
            +1 mvninstall 14m 57s trunk passed
            +1 compile 7m 21s trunk passed
            +1 checkstyle 0m 54s trunk passed
            +1 mvnsite 1m 13s trunk passed
            +1 shadedclient 10m 57s branch has no errors when building and testing our client artifacts.
            -1 findbugs 1m 8s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 0m 53s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 10s Maven dependency ordering for patch
            +1 mvninstall 1m 5s the patch passed
            +1 compile 8m 12s the patch passed
            +1 javac 8m 12s the patch passed
            -0 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415)
            +1 mvnsite 1m 12s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 9m 29s patch has no errors when building and testing our client artifacts.
            +1 findbugs 2m 25s the patch passed
            +1 javadoc 0m 51s the patch passed
                  Other Tests
            +1 unit 0m 37s hadoop-yarn-api in the patch passed.
            -1 unit 76m 35s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 28s The patch does not generate ASF License warnings.
            140m 29s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7738
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 7dd03ab48ebf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / b278f7b
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19224/testReport/
            Max. process+thread count 896 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/19224/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 14m 57s trunk passed +1 compile 7m 21s trunk passed +1 checkstyle 0m 54s trunk passed +1 mvnsite 1m 13s trunk passed +1 shadedclient 10m 57s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 8s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 53s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 5s the patch passed +1 compile 8m 12s the patch passed +1 javac 8m 12s the patch passed -0 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) +1 mvnsite 1m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 9m 29s patch has no errors when building and testing our client artifacts. +1 findbugs 2m 25s the patch passed +1 javadoc 0m 51s the patch passed       Other Tests +1 unit 0m 37s hadoop-yarn-api in the patch passed. -1 unit 76m 35s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 140m 29s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 7dd03ab48ebf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / b278f7b maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19224/testReport/ Max. process+thread count 896 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/19224/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 9m 44s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 9s Maven dependency ordering for branch
            +1 mvninstall 16m 51s trunk passed
            +1 compile 7m 56s trunk passed
            +1 checkstyle 1m 6s trunk passed
            +1 mvnsite 1m 31s trunk passed
            +1 shadedclient 12m 39s branch has no errors when building and testing our client artifacts.
            -1 findbugs 1m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 0m 58s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 12s Maven dependency ordering for patch
            +1 mvninstall 1m 19s the patch passed
            +1 compile 7m 0s the patch passed
            +1 javac 7m 0s the patch passed
            -0 checkstyle 1m 2s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415)
            +1 mvnsite 1m 21s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 11m 1s patch has no errors when building and testing our client artifacts.
            +1 findbugs 2m 47s the patch passed
            +1 javadoc 0m 56s the patch passed
                  Other Tests
            +1 unit 0m 40s hadoop-yarn-api in the patch passed.
            -1 unit 62m 30s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 34s The patch does not generate ASF License warnings.
            142m 0s



            Reason Tests
            Failed junit tests hadoop.yarn.server.resourcemanager.TestLeaderElectorService



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7738
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux e380e3d87536 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 3d65dbe
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19227/testReport/
            Max. process+thread count 866 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/19227/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 9m 44s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 16m 51s trunk passed +1 compile 7m 56s trunk passed +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 31s trunk passed +1 shadedclient 12m 39s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 58s trunk passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 7m 0s the patch passed +1 javac 7m 0s the patch passed -0 checkstyle 1m 2s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) +1 mvnsite 1m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 1s patch has no errors when building and testing our client artifacts. +1 findbugs 2m 47s the patch passed +1 javadoc 0m 56s the patch passed       Other Tests +1 unit 0m 40s hadoop-yarn-api in the patch passed. -1 unit 62m 30s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 142m 0s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestLeaderElectorService Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7738 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux e380e3d87536 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 3d65dbe maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/19227/testReport/ Max. process+thread count 866 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/19227/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            leftnoteasy Wangda Tan added a comment -

            sunilg could u help check the latest patch? Test failure should not related.

            leftnoteasy Wangda Tan added a comment - sunilg could u help check the latest patch? Test failure should not related.
            sunilg Sunil G added a comment -

            +1 Committing shortly.

            sunilg Sunil G added a comment - +1 Committing shortly.
            sunilg Sunil G added a comment - - edited

            Committed to trunk. wangda, is it needed to commit to branch-3.0. I find patch is not applying. Could you please help to check this, i found TestCapacitySchedulerWithMultiResourceTypes is not preset in branch-3.0

            sunilg Sunil G added a comment - - edited Committed to trunk. wangda , is it needed to commit to branch-3.0. I find patch is not applying. Could you please help to check this, i found TestCapacitySchedulerWithMultiResourceTypes is not preset in branch-3.0
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13517 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13517/)
            YARN-7738. CapacityScheduler: Support refresh maximum allocation for (sunilg: rev 06cceba1cb07340c412c4467439c16ea6812e685)

            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceUtils.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerWithMultiResourceTypes.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13517 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13517/ ) YARN-7738 . CapacityScheduler: Support refresh maximum allocation for (sunilg: rev 06cceba1cb07340c412c4467439c16ea6812e685) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceUtils.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerWithMultiResourceTypes.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
            sunilg Sunil G added a comment -

            Closing for 3.1.0. We will check whether this is needed for 3.0

            sunilg Sunil G added a comment - Closing for 3.1.0. We will check whether this is needed for 3.0
            xiangli Xiang Li added a comment - - edited

            Hi leftnoteasy sunilg, it seems we hits a similar issue on our production system (Hadoop 2.7.3). Could we have a patch for branch-2 (or branch-2.7.3) ?
            I am trying to back-port the patch to branch-2.7.3, but the patch is based on resource-types and ResourceUtils, which are not available in branch-2.

            xiangli Xiang Li added a comment - - edited Hi leftnoteasy sunilg , it seems we hits a similar issue on our production system (Hadoop 2.7.3). Could we have a patch for branch-2 (or branch-2.7.3) ? I am trying to back-port the patch to branch-2.7.3, but the patch is based on resource-types and ResourceUtils, which are not available in branch-2.
            leftnoteasy Wangda Tan added a comment -

            water This is not an issue exists in 2.7.3, this is a 3.0/3.1 issue.

            leftnoteasy Wangda Tan added a comment - water This is not an issue exists in 2.7.3, this is a 3.0/3.1 issue.
            xiangli Xiang Li added a comment -

            leftnoteasy thanks for taking care of this! I see. I will study it and get back here if I could have more.

            xiangli Xiang Li added a comment - leftnoteasy thanks for taking care of this! I see. I will study it and get back here if I could have more.
            yufeigu Yufei Gu added a comment - - edited

            leftnoteasy, could you tell me a little more about why CS supports refresh maximum allocation while RM is running? It looks kind of weird to me that reload the whole setting in resource-type.xml isn't supported, but only one of these setting is allowed to reload dynamically. Thanks.

            yufeigu Yufei Gu added a comment - - edited leftnoteasy , could you tell me a little more about why CS supports refresh maximum allocation while RM is running? It looks kind of weird to me that reload the whole setting in resource-type.xml isn't supported, but only one of these setting is allowed to reload dynamically. Thanks.
            leftnoteasy Wangda Tan added a comment -

            yufeigu, CS support refresh maximum allocation while RM is running, and the new_max_alloc must >= old_max_alloc.

            The patch fixed an issue that resource-types.xml isn't loaded during the refresh. This causes after refresh, all maximum-allocation for non-memory/vcore resource becomes 0

            leftnoteasy Wangda Tan added a comment - yufeigu , CS support refresh maximum allocation while RM is running, and the new_max_alloc must >= old_max_alloc. The patch fixed an issue that resource-types.xml isn't loaded during the refresh. This causes after refresh, all maximum-allocation for non-memory/vcore resource becomes 0
            yufeigu Yufei Gu added a comment -

            Thanks leftnoteasy. If that is so, is it reasonable to make sure resource-types.xml to be loaded before any scheduler refresh? So that we don't need to reload maximum allocation any more.

            yufeigu Yufei Gu added a comment - Thanks leftnoteasy . If that is so, is it reasonable to make sure resource-types.xml to be loaded before any scheduler refresh? So that we don't need to reload maximum allocation any more.

            People

              leftnoteasy Wangda Tan
              ssathish@hortonworks.com Sumana Sathish
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: