Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      RM side changes to allow an AM to ask for change of ExecutionType.

      Currently, there are two cases:

      1. Promotion : OPPORTUNISTIC to GUARANTEED.
      2. Demotion : GUARANTEED to OPPORTUNISTIC.

      This is similar to YARN-1197 which allows for change in Container resources.

      1. YARN-5959.005.patch
        190 kB
        Arun Suresh
      2. YARN-5959.combined.001.patch
        152 kB
        Arun Suresh
      3. YARN-5959.wip.002.patch
        77 kB
        Arun Suresh
      4. YARN-5959.wip.003.patch
        107 kB
        Arun Suresh
      5. YARN-5959.wip.patch
        81 kB
        Arun Suresh
      6. YARN-5959-branch-2.001.patch
        185 kB
        Arun Suresh
      7. YARN-5959-branch-2.002.patch
        185 kB
        Arun Suresh
      8. YARN-5959-YARN-5085.001.patch
        110 kB
        Arun Suresh
      9. YARN-5959-YARN-5085.002.patch
        110 kB
        Arun Suresh
      10. YARN-5959-YARN-5085.003.patch
        166 kB
        Arun Suresh
      11. YARN-5959-YARN-5085.004.patch
        187 kB
        Arun Suresh
      12. YARN-5959-YARN-5085.005.patch
        190 kB
        Arun Suresh

        Activity

        Hide
        asuresh Arun Suresh added a comment - - edited

        This can actually be accomplished in a scheduler agnostic manner.

        Promotion can be implemented as follows:

        1. Create a dummy ResourceRequest with the same resource, priority etc. of the existing opportunistic container
        2. Set the required resource name (target location) to the node on which in the original container was allocated.
        3. Set relaxLocality to false.
        4. Set the allocationRequestId of the request to be negative, so it will be considered before any other ResourceRequest with the same priority.
        5. Once the scheduler allocates an RMContainer against the dummy request, swap the containerIds with the existing container and release this new container. This will ensure that the resource accounting is correct. The allocation path ensures that available resources in the queue, application and node resources are correctly decremented and the release ensures that only the opportunistic resource (essentially 0 resource container from the perspective of the scheduler) are reclaimed.

        Demotion will happen in exactly the reverse:
        A dummy OPPORTUNISTIC container is created to swap with the existing GUARANTEED container on the same node. The containerIds are then swapped and the dummy container is release ensuring guaranteed resources are reclaimed.

        Show
        asuresh Arun Suresh added a comment - - edited This can actually be accomplished in a scheduler agnostic manner. Promotion can be implemented as follows: Create a dummy ResourceRequest with the same resource, priority etc. of the existing opportunistic container Set the required resource name (target location) to the node on which in the original container was allocated. Set relaxLocality to false . Set the allocationRequestId of the request to be negative, so it will be considered before any other ResourceRequest with the same priority. Once the scheduler allocates an RMContainer against the dummy request, swap the containerIds with the existing container and release this new container. This will ensure that the resource accounting is correct. The allocation path ensures that available resources in the queue, application and node resources are correctly decremented and the release ensures that only the opportunistic resource (essentially 0 resource container from the perspective of the scheduler) are reclaimed. Demotion will happen in exactly the reverse: A dummy OPPORTUNISTIC container is created to swap with the existing GUARANTEED container on the same node. The containerIds are then swapped and the dummy container is release ensuring guaranteed resources are reclaimed.
        Hide
        asuresh Arun Suresh added a comment - - edited

        Attaching a WIP patch with tests to demonstrate the proposal in the previous comment works
        Ping : Wangda Tan, Karthik Kambatla, Subru Krishnan, Carlo Curino, Konstantinos Karanasos, do take a look

        The only really interesting changes are in the SchedulerAplicationAttempt class. Most of the remaining changes are related to wiring.

        Show
        asuresh Arun Suresh added a comment - - edited Attaching a WIP patch with tests to demonstrate the proposal in the previous comment works Ping : Wangda Tan , Karthik Kambatla , Subru Krishnan , Carlo Curino , Konstantinos Karanasos , do take a look The only really interesting changes are in the SchedulerAplicationAttempt class. Most of the remaining changes are related to wiring.
        Hide
        asuresh Arun Suresh added a comment -

        Since the WIP patch depends on YARN-5938, am attaching a combined patch that applies against trunk to kick off Jenkins

        Show
        asuresh Arun Suresh added a comment - Since the WIP patch depends on YARN-5938 , am attaching a combined patch that applies against trunk to kick off Jenkins
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 14 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 7m 7s trunk passed
        +1 compile 5m 6s trunk passed
        +1 checkstyle 1m 6s trunk passed
        +1 mvnsite 3m 33s trunk passed
        +1 mvneclipse 2m 0s trunk passed
        +1 findbugs 5m 36s trunk passed
        +1 javadoc 2m 34s trunk passed
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 2m 31s the patch passed
        +1 compile 4m 51s the patch passed
        +1 javac 4m 51s the patch passed
        -0 checkstyle 1m 8s hadoop-yarn-project/hadoop-yarn: The patch generated 32 new + 1555 unchanged - 16 fixed = 1587 total (was 1571)
        +1 mvnsite 3m 28s the patch passed
        +1 mvneclipse 1m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        -1 findbugs 1m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 2m 35s the patch passed
        +1 unit 0m 32s hadoop-yarn-api in the patch passed.
        +1 unit 2m 35s hadoop-yarn-common in the patch passed.
        +1 unit 0m 33s hadoop-yarn-server-common in the patch passed.
        +1 unit 13m 52s hadoop-yarn-server-nodemanager in the patch passed.
        -1 unit 43m 49s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 unit 16m 43s hadoop-yarn-client in the patch failed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        138m 24s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          org.apache.hadoop.yarn.server.scheduler.OpportunisticContainerContext.UNDEFINED isn't final but should be At OpportunisticContainerContext.java:be At OpportunisticContainerContext.java:[line 58]
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.getContainer() is unsynchronized, org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.setContainer(Container) is synchronized At RMContainerImpl.java:synchronized At RMContainerImpl.java:[line 280]
        Failed junit tests hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService
          hadoop.yarn.client.api.impl.TestAMRMClient



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841624/YARN-5959.combined.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7e75f090fc22 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f885160
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14178/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14178/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 14 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 5m 6s trunk passed +1 checkstyle 1m 6s trunk passed +1 mvnsite 3m 33s trunk passed +1 mvneclipse 2m 0s trunk passed +1 findbugs 5m 36s trunk passed +1 javadoc 2m 34s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 2m 31s the patch passed +1 compile 4m 51s the patch passed +1 javac 4m 51s the patch passed -0 checkstyle 1m 8s hadoop-yarn-project/hadoop-yarn: The patch generated 32 new + 1555 unchanged - 16 fixed = 1587 total (was 1571) +1 mvnsite 3m 28s the patch passed +1 mvneclipse 1m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 findbugs 1m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 35s the patch passed +1 unit 0m 32s hadoop-yarn-api in the patch passed. +1 unit 2m 35s hadoop-yarn-common in the patch passed. +1 unit 0m 33s hadoop-yarn-server-common in the patch passed. +1 unit 13m 52s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 43m 49s hadoop-yarn-server-resourcemanager in the patch failed. -1 unit 16m 43s hadoop-yarn-client in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 138m 24s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common   org.apache.hadoop.yarn.server.scheduler.OpportunisticContainerContext.UNDEFINED isn't final but should be At OpportunisticContainerContext.java:be At OpportunisticContainerContext.java: [line 58] FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.getContainer() is unsynchronized, org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.setContainer(Container) is synchronized At RMContainerImpl.java:synchronized At RMContainerImpl.java: [line 280] Failed junit tests hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService   hadoop.yarn.client.api.impl.TestAMRMClient Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841624/YARN-5959.combined.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7e75f090fc22 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f885160 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14178/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14178/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14178/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Moved out the AMRMClient changes to YARN-5966 and added more tests.

        Show
        asuresh Arun Suresh added a comment - Moved out the AMRMClient changes to YARN-5966 and added more tests.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 7s YARN-5959 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841803/YARN-5959.wip.002.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14187/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s YARN-5959 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841803/YARN-5959.wip.002.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/14187/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hrsharma Hitesh Sharma added a comment -

        Hello Arun Suresh, can you share some design around how NM handles the change in execution type? I will look at the patch more closely but having that context in mind will help.

        Thanks a lot!

        Show
        hrsharma Hitesh Sharma added a comment - Hello Arun Suresh , can you share some design around how NM handles the change in execution type? I will look at the patch more closely but having that context in mind will help. Thanks a lot!
        Hide
        asuresh Arun Suresh added a comment -

        Hitesh Sharma, this patch address only the RM side changes. The changes to the NM will be handled in YARN-5977 and YARN-5978

        Show
        asuresh Arun Suresh added a comment - Hitesh Sharma , this patch address only the RM side changes. The changes to the NM will be handled in YARN-5977 and YARN-5978
        Hide
        leftnoteasy Wangda Tan added a comment -

        Look at overall changes and discussed with Arun Suresh offline.

        A couple of things we discussed:
        1) Currently approach in this patch is, if any normal container request with higher priority, it will be allocated first. However this is discussable: if we give the available resource of a node to a different normal container, it is possible that the running opportunistic container will be killed and work will be wasted.
        2) Instead of creating a hard-locality container request, can we wrap the ResourceRequest to add some additional information to identify it is for container increase / promotion. This could be useful for:

        • Track pending container changes requests.
        • Cancel / update pending container change requests
        • Makes backend implementation of container increase / promotion to be same.
        • If you think it sounds like a plan, I can also help with working on refactoring patch / reviews.
        Show
        leftnoteasy Wangda Tan added a comment - Look at overall changes and discussed with Arun Suresh offline. A couple of things we discussed: 1) Currently approach in this patch is, if any normal container request with higher priority, it will be allocated first. However this is discussable: if we give the available resource of a node to a different normal container, it is possible that the running opportunistic container will be killed and work will be wasted. 2) Instead of creating a hard-locality container request, can we wrap the ResourceRequest to add some additional information to identify it is for container increase / promotion. This could be useful for: Track pending container changes requests. Cancel / update pending container change requests Makes backend implementation of container increase / promotion to be same. If you think it sounds like a plan, I can also help with working on refactoring patch / reviews.
        Hide
        asuresh Arun Suresh added a comment - - edited

        Thanks for capturing the salient points from our discussion Wangda Tan.
        Updating patch based your feedback..

        1.

        Instead of creating a hard-locality container request, can we wrap the ResourceRequest to add some additional information to identify it is for container increase / promotion.

        It looks like we already have an AbstractResourceRequest which both the ResourceRequest and UpdateContainerRequest extends from.
        I renamed it to SchedulerResourceRequest and also included a field containerToUpdate to signify this is an update to an existing container.

        2.

        if any normal container request with higher priority, it will be allocated first. However this is discussable: if we give the available resource of a node to a different normal container, it is possible that the running opportunistic container will be killed and work will be wasted.

        This is IMO still an ordering issue. I have updated the SchedulerRequestKey to ensure the behavior you pointed out. Update requests (both promotion as well as resource updates) will be considered before any normal resourceRequest. Since the schedulerKey is created using a SchedulerResourceRequest, which contains information about the container update and not the old ResourceRequest, I do not have to hard code a negative allocationRequestId as I had done in the previous patch.

        3.

        • Track pending container changes requests.
        • Cancel / update pending container change requests
        • Makes backend implementation of container increase / promotion to be same.

        I feel all these cases can be taken care of in the new patch by ensuring that the SchedulerResourceRequest and SchedulerRequestKey understands that it is pertaining to a container update.
        Once you agree on this new design, I can go ahead (I think we can do that in a separate JIRA) and create a patch that unifies the backend for container resource increase and promotion.

        Show
        asuresh Arun Suresh added a comment - - edited Thanks for capturing the salient points from our discussion Wangda Tan . Updating patch based your feedback.. 1. Instead of creating a hard-locality container request, can we wrap the ResourceRequest to add some additional information to identify it is for container increase / promotion. It looks like we already have an AbstractResourceRequest which both the ResourceRequest and UpdateContainerRequest extends from. I renamed it to SchedulerResourceRequest and also included a field containerToUpdate to signify this is an update to an existing container. 2. if any normal container request with higher priority, it will be allocated first. However this is discussable: if we give the available resource of a node to a different normal container, it is possible that the running opportunistic container will be killed and work will be wasted. This is IMO still an ordering issue. I have updated the SchedulerRequestKey to ensure the behavior you pointed out. Update requests (both promotion as well as resource updates) will be considered before any normal resourceRequest. Since the schedulerKey is created using a SchedulerResourceRequest , which contains information about the container update and not the old ResourceRequest , I do not have to hard code a negative allocationRequestId as I had done in the previous patch. 3. Track pending container changes requests. Cancel / update pending container change requests Makes backend implementation of container increase / promotion to be same. I feel all these cases can be taken care of in the new patch by ensuring that the SchedulerResourceRequest and SchedulerRequestKey understands that it is pertaining to a container update. Once you agree on this new design, I can go ahead (I think we can do that in a separate JIRA) and create a patch that unifies the backend for container resource increase and promotion.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 5s YARN-5959 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842846/YARN-5959.wip.003.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14261/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-5959 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842846/YARN-5959.wip.003.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/14261/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Cancelling the patch... since it does not apply without YARN-5938.
        YARN-5085 has the combined patch

        Show
        asuresh Arun Suresh added a comment - Cancelling the patch... since it does not apply without YARN-5938 . YARN-5085 has the combined patch
        Hide
        asuresh Arun Suresh added a comment -

        Based on offline discussions with Wangda Tan, I will rework this patch to refactor the internal RM implementation of container increase-decrease based on the approach outlined here to unify the codebase.

        Show
        asuresh Arun Suresh added a comment - Based on offline discussions with Wangda Tan , I will rework this patch to refactor the internal RM implementation of container increase-decrease based on the approach outlined here to unify the codebase.
        Hide
        asuresh Arun Suresh added a comment -

        Updating patch.

        • The container increase and decreases accounting has been moved to the AppSchedulingInfo.
        • To isolate the update code path (and to make it easier to review), i've created a ContainerUpdateContext which is field of the AppSchedulingInfo. It also encapsulates the datastructures used for accounting the container increase and decrease.

        The new functionality can be used for both ExecutionType Promotion/Demotion as well as Container resource Increase/Decrease. I have currently just hooked it to the ExecutionType promote and demote. I feel we should raise another JIRA to work on removing the existing container increase/decrease code-path and move it to using the new ContainerUpdateContext.

        As with the previous patches, the basic principle is the same.. new ResourceRequests are created for container increases, but these are created internally in the AppSchedulingInfo

        Wangda Tan, do give it a look.

        Show
        asuresh Arun Suresh added a comment - Updating patch. The container increase and decreases accounting has been moved to the AppSchedulingInfo . To isolate the update code path (and to make it easier to review), i've created a ContainerUpdateContext which is field of the AppSchedulingInfo . It also encapsulates the datastructures used for accounting the container increase and decrease. The new functionality can be used for both ExecutionType Promotion/Demotion as well as Container resource Increase/Decrease. I have currently just hooked it to the ExecutionType promote and demote. I feel we should raise another JIRA to work on removing the existing container increase/decrease code-path and move it to using the new ContainerUpdateContext. As with the previous patches, the basic principle is the same.. new ResourceRequests are created for container increases, but these are created internally in the AppSchedulingInfo Wangda Tan , do give it a look.
        Hide
        asuresh Arun Suresh added a comment -

        Correcting name of patch file to kick of Jenkins against YARN-5085 branch.

        Show
        asuresh Arun Suresh added a comment - Correcting name of patch file to kick of Jenkins against YARN-5085 branch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
        0 mvndep 0m 53s Maven dependency ordering for branch
        +1 mvninstall 12m 58s YARN-5085 passed
        +1 compile 4m 52s YARN-5085 passed
        +1 checkstyle 0m 49s YARN-5085 passed
        +1 mvnsite 1m 37s YARN-5085 passed
        +1 mvneclipse 0m 56s YARN-5085 passed
        +1 findbugs 2m 55s YARN-5085 passed
        +1 javadoc 1m 11s YARN-5085 passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 12s the patch passed
        +1 compile 4m 35s the patch passed
        +1 javac 4m 35s the patch passed
        -0 checkstyle 0m 47s hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 455 unchanged - 4 fixed = 475 total (was 459)
        +1 mvnsite 1m 32s the patch passed
        +1 mvneclipse 0m 54s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        -1 findbugs 1m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
        +1 javadoc 1m 7s the patch passed
        +1 unit 0m 31s hadoop-yarn-api in the patch passed.
        +1 unit 0m 31s hadoop-yarn-server-common in the patch passed.
        +1 unit 39m 13s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 32s The patch does not generate ASF License warnings.
        89m 7s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          32 bit int shifted by 32 bits in org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey.hashCode() At SchedulerRequestKey.java:32 bits in org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey.hashCode() At SchedulerRequestKey.java:[line 145]
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.getContainer() is unsynchronized, org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.setContainer(Container) is synchronized At RMContainerImpl.java:synchronized At RMContainerImpl.java:[line 280]
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.ContainerUpdateContext.UNDEFINED isn't final but should be At ContainerUpdateContext.java:be At ContainerUpdateContext.java:[line 49]



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845261/YARN-5959-YARN-5085.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 04a2a17659e9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5085 / 6938b67
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14514/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14514/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14514/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14514/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14514/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 12m 58s YARN-5085 passed +1 compile 4m 52s YARN-5085 passed +1 checkstyle 0m 49s YARN-5085 passed +1 mvnsite 1m 37s YARN-5085 passed +1 mvneclipse 0m 56s YARN-5085 passed +1 findbugs 2m 55s YARN-5085 passed +1 javadoc 1m 11s YARN-5085 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 4m 35s the patch passed +1 javac 4m 35s the patch passed -0 checkstyle 0m 47s hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 455 unchanged - 4 fixed = 475 total (was 459) +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 findbugs 1m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 1m 7s the patch passed +1 unit 0m 31s hadoop-yarn-api in the patch passed. +1 unit 0m 31s hadoop-yarn-server-common in the patch passed. +1 unit 39m 13s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 89m 7s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common   32 bit int shifted by 32 bits in org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey.hashCode() At SchedulerRequestKey.java:32 bits in org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey.hashCode() At SchedulerRequestKey.java: [line 145] FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.getContainer() is unsynchronized, org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.setContainer(Container) is synchronized At RMContainerImpl.java:synchronized At RMContainerImpl.java: [line 280]   org.apache.hadoop.yarn.server.resourcemanager.scheduler.ContainerUpdateContext.UNDEFINED isn't final but should be At ContainerUpdateContext.java:be At ContainerUpdateContext.java: [line 49] Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845261/YARN-5959-YARN-5085.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 04a2a17659e9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5085 / 6938b67 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14514/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14514/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14514/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14514/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14514/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Fixing findbugs.

        Show
        asuresh Arun Suresh added a comment - Fixing findbugs.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        -1 mvninstall 9m 52s root in YARN-5085 failed.
        +1 compile 4m 52s YARN-5085 passed
        +1 checkstyle 0m 50s YARN-5085 passed
        +1 mvnsite 1m 37s YARN-5085 passed
        +1 mvneclipse 0m 57s YARN-5085 passed
        +1 findbugs 2m 52s YARN-5085 passed
        +1 javadoc 1m 10s YARN-5085 passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 12s the patch passed
        +1 compile 4m 33s the patch passed
        +1 javac 4m 33s the patch passed
        -0 checkstyle 0m 48s hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 455 unchanged - 4 fixed = 473 total (was 459)
        +1 mvnsite 1m 33s the patch passed
        +1 mvneclipse 0m 54s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 12s the patch passed
        +1 javadoc 1m 8s the patch passed
        +1 unit 0m 31s hadoop-yarn-api in the patch passed.
        +1 unit 0m 32s hadoop-yarn-server-common in the patch passed.
        -1 unit 38m 53s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        84m 56s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845275/YARN-5959-YARN-5085.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a1da828b7065 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5085 / 6938b67
        Default Java 1.8.0_111
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14517/artifact/patchprocess/branch-mvninstall-root.txt
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14517/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14517/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14517/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14517/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch -1 mvninstall 9m 52s root in YARN-5085 failed. +1 compile 4m 52s YARN-5085 passed +1 checkstyle 0m 50s YARN-5085 passed +1 mvnsite 1m 37s YARN-5085 passed +1 mvneclipse 0m 57s YARN-5085 passed +1 findbugs 2m 52s YARN-5085 passed +1 javadoc 1m 10s YARN-5085 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 4m 33s the patch passed +1 javac 4m 33s the patch passed -0 checkstyle 0m 48s hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 455 unchanged - 4 fixed = 473 total (was 459) +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 12s the patch passed +1 javadoc 1m 8s the patch passed +1 unit 0m 31s hadoop-yarn-api in the patch passed. +1 unit 0m 32s hadoop-yarn-server-common in the patch passed. -1 unit 38m 53s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 84m 56s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845275/YARN-5959-YARN-5085.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a1da828b7065 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5085 / 6938b67 Default Java 1.8.0_111 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14517/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14517/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14517/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14517/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14517/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Arun Suresh for updating the patch.

        I think the latest patch is much better than the previous one, but I found we can merge logics of execution type update and resizing even more. And we can make call flow of allocation more clear:

        High level comments:

        1) Can we uniform call flow of OpportunisticContainerAllocatorAMService (OCAAMS) and normal AMS?
        Currently, OCAAMS gets SchedulerApplicationAttempt, and then directly update request to ContainerUpdateContext. And for increase/decrease request, they will be added to Scheduler.allocate, and set through AppSchedulingInfo#updateResourceRequests.

        I think the two could be merged: Since we have a unified field of UpdateContainerRequest inside AllocateRequest, and we calls RMServerUtils#validateAndSplitUpdateResourceRequests from ApplicationMasterService, so we should be able to split increase/decrease/demote/promote requests. Then increase / promote requests will be converted to normal resource requests and add to AppSchedulingInfo (via Scheduler#allocate).

        Inside AppSchedulingInfo, it could uses ContainerUpdateContext to maintain all container-update-related requests.

        2) Similar to above, it's better to unify call flow of fetch container update result as well:

        Currently it calls appAttempt.pullContainersWithUpdatedExecType, I think it should be merged to FiCaSchedulerApp#getAllocation. And actually, we should remove increasedContainers/decreasedContainers, and instead we should have: List<Entry<ContainerUpdateType, List<Container>> to capture all kinds of container updates.

        Summary of my opinions about call flow:

        a. Send update container request to scheduler:
        For increase/decrease container: Inside ApplicationMasterService, calls scheduler#allocate (same as today)
        For promotion/demotion container:
        OCAAMS split request to two parts:

        • List of opportunistic container allocations: opportunistic allocations Requests goes to OpportunisticContainerAllocator (same as today)
        • UpdateContainerRequests (promotion and demotion) goes to ApplicationMasterService

        Minor refactoring may require for Scheduler#allocate's parameters, since we can merge all container update requests to a single one. like List<Entry<ContainerUpdateType, List<ContainerUpdateRequest>>.

        b. Get update container response from scheduler:
        For both increase/decrease/promotion/demotion containers, they will be set to Allocation by SchedulerApplicationAttempt implementations.

        Other suggestions
        3) Can we avoid changing AbstractResourceRequest (assume we may remove it on YARN-6022)? Adding a updatedContainerId to ResourceRequest is very confusing to application, because they may set ResourceRequest#updatedContainerId to update a container, but it won't work.
        I believe we should have enough context (like the ContainerId inside SchedulerRequestKey) already, could you reconsider this part?

        4) ContainerUpdateType: It's better to have separated promotion / demotion instead of a single "update-execution-type".

        Thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Arun Suresh for updating the patch. I think the latest patch is much better than the previous one, but I found we can merge logics of execution type update and resizing even more. And we can make call flow of allocation more clear: High level comments: 1) Can we uniform call flow of OpportunisticContainerAllocatorAMService (OCAAMS) and normal AMS? Currently, OCAAMS gets SchedulerApplicationAttempt, and then directly update request to ContainerUpdateContext. And for increase/decrease request, they will be added to Scheduler.allocate, and set through AppSchedulingInfo#updateResourceRequests. I think the two could be merged: Since we have a unified field of UpdateContainerRequest inside AllocateRequest, and we calls RMServerUtils#validateAndSplitUpdateResourceRequests from ApplicationMasterService, so we should be able to split increase/decrease/demote/promote requests. Then increase / promote requests will be converted to normal resource requests and add to AppSchedulingInfo (via Scheduler#allocate). Inside AppSchedulingInfo, it could uses ContainerUpdateContext to maintain all container-update-related requests. 2) Similar to above, it's better to unify call flow of fetch container update result as well: Currently it calls appAttempt.pullContainersWithUpdatedExecType, I think it should be merged to FiCaSchedulerApp#getAllocation. And actually, we should remove increasedContainers/decreasedContainers, and instead we should have: List<Entry<ContainerUpdateType, List<Container>> to capture all kinds of container updates. Summary of my opinions about call flow: a. Send update container request to scheduler: For increase/decrease container: Inside ApplicationMasterService, calls scheduler#allocate (same as today) For promotion/demotion container: OCAAMS split request to two parts: List of opportunistic container allocations: opportunistic allocations Requests goes to OpportunisticContainerAllocator (same as today) UpdateContainerRequests (promotion and demotion) goes to ApplicationMasterService Minor refactoring may require for Scheduler#allocate's parameters, since we can merge all container update requests to a single one. like List<Entry<ContainerUpdateType, List<ContainerUpdateRequest>> . b. Get update container response from scheduler: For both increase/decrease/promotion/demotion containers, they will be set to Allocation by SchedulerApplicationAttempt implementations. Other suggestions 3) Can we avoid changing AbstractResourceRequest (assume we may remove it on YARN-6022 )? Adding a updatedContainerId to ResourceRequest is very confusing to application, because they may set ResourceRequest#updatedContainerId to update a container, but it won't work. I believe we should have enough context (like the ContainerId inside SchedulerRequestKey) already, could you reconsider this part? 4) ContainerUpdateType: It's better to have separated promotion / demotion instead of a single "update-execution-type". Thoughts?
        Hide
        asuresh Arun Suresh added a comment -

        Updating patch.

        Thanks for the detailed review Wangda Tan.
        I agree with all your suggestions. I've updated the patch as follows:

        1. Reverted all changes to the AbstractResourceRequest. We don't need the extra containerToUpdate field anymore, as you pointed out, it was possible to accomplish everything via the SchedulerRequestKey
        2. Modified the call flow remove the update Execution type call from the OCAAMS and moved it into the Scheduler itself.
        3. Modified the Allocation object to handle the promoted and demoted containers.
        4. Split the ContainerUpdateType::UPDATE_EXECUTION_TYPE to PROMOTE_EXECUTION_TYPE and DEMOTE_EXECUTION_TYPE.

        The size of the patch has increased a bit since I modified the allocate() method of the scheduler to take the promoted and demoted lists. And this touches a few test classes.

        Show
        asuresh Arun Suresh added a comment - Updating patch. Thanks for the detailed review Wangda Tan . I agree with all your suggestions. I've updated the patch as follows: Reverted all changes to the AbstractResourceRequest . We don't need the extra containerToUpdate field anymore, as you pointed out, it was possible to accomplish everything via the SchedulerRequestKey Modified the call flow remove the update Execution type call from the OCAAMS and moved it into the Scheduler itself. Modified the Allocation object to handle the promoted and demoted containers. Split the ContainerUpdateType::UPDATE_EXECUTION_TYPE to PROMOTE_EXECUTION_TYPE and DEMOTE_EXECUTION_TYPE. The size of the patch has increased a bit since I modified the allocate() method of the scheduler to take the promoted and demoted lists. And this touches a few test classes.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 15 new or modified test files.
        0 mvndep 0m 11s Maven dependency ordering for branch
        -1 mvninstall 11m 47s root in YARN-5085 failed.
        +1 compile 5m 39s YARN-5085 passed
        +1 checkstyle 1m 5s YARN-5085 passed
        +1 mvnsite 2m 23s YARN-5085 passed
        +1 mvneclipse 1m 25s YARN-5085 passed
        +1 findbugs 4m 17s YARN-5085 passed
        +1 javadoc 1m 45s YARN-5085 passed
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 55s the patch passed
        +1 compile 5m 17s the patch passed
        +1 cc 5m 17s the patch passed
        +1 javac 5m 17s the patch passed
        -0 checkstyle 1m 10s hadoop-yarn-project/hadoop-yarn: The patch generated 39 new + 1421 unchanged - 17 fixed = 1460 total (was 1438)
        +1 mvnsite 2m 10s the patch passed
        +1 mvneclipse 1m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 53s the patch passed
        -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 3 new + 913 unchanged - 0 fixed = 916 total (was 913)
        +1 unit 0m 36s hadoop-yarn-api in the patch passed.
        +1 unit 0m 35s hadoop-yarn-server-common in the patch passed.
        +1 unit 42m 7s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 unit 17m 2s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        117m 3s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845558/YARN-5959-YARN-5085.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 34744e0833bc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5085 / 6938b67
        Default Java 1.8.0_111
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14549/artifact/patchprocess/branch-mvninstall-root.txt
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14549/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14549/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14549/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14549/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 15 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch -1 mvninstall 11m 47s root in YARN-5085 failed. +1 compile 5m 39s YARN-5085 passed +1 checkstyle 1m 5s YARN-5085 passed +1 mvnsite 2m 23s YARN-5085 passed +1 mvneclipse 1m 25s YARN-5085 passed +1 findbugs 4m 17s YARN-5085 passed +1 javadoc 1m 45s YARN-5085 passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 55s the patch passed +1 compile 5m 17s the patch passed +1 cc 5m 17s the patch passed +1 javac 5m 17s the patch passed -0 checkstyle 1m 10s hadoop-yarn-project/hadoop-yarn: The patch generated 39 new + 1421 unchanged - 17 fixed = 1460 total (was 1438) +1 mvnsite 2m 10s the patch passed +1 mvneclipse 1m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 53s the patch passed -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 3 new + 913 unchanged - 0 fixed = 916 total (was 913) +1 unit 0m 36s hadoop-yarn-api in the patch passed. +1 unit 0m 35s hadoop-yarn-server-common in the patch passed. +1 unit 42m 7s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 17m 2s hadoop-yarn-client in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 117m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845558/YARN-5959-YARN-5085.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 34744e0833bc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5085 / 6938b67 Default Java 1.8.0_111 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14549/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14549/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14549/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14549/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14549/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Arun Suresh for update the patch, generally looks good, some additional minor suggestions:

        1) Can we make scheduler.allocate to have a single ContainerUpdateRequests, which include 4 lists (increase/decrease/promotion/demotion) instead of making all of them become top-level parameter, similarly for Allocation.

        2) AMS: can we move 4 invokes of addToUpdatedContainers and 1 addToUpdateContainerErrors to a separate method considering the long method.

        3) Remove unused imports of OCAAMS

        4) There's a duplicated method createRmContainer in AbstractYarnScheduler and OCAAMS, could we:

        • Move it to SchedulerUtils?
        • And in addition, it will only allocate OpportunisticContainer, so could you rename to createOpportuniscRMContainer. I'm asking this because it cannot guarantee atomic of app and node, I want to make sure no guaranteed container allocation uses this method.

        5) RMContainerImpl:

        • Use volatile to avoid synchronized lock for set/getContainer? Because all other methods are using R/W lock.
        Show
        leftnoteasy Wangda Tan added a comment - Thanks Arun Suresh for update the patch, generally looks good, some additional minor suggestions: 1) Can we make scheduler.allocate to have a single ContainerUpdateRequests, which include 4 lists (increase/decrease/promotion/demotion) instead of making all of them become top-level parameter, similarly for Allocation. 2) AMS: can we move 4 invokes of addToUpdatedContainers and 1 addToUpdateContainerErrors to a separate method considering the long method. 3) Remove unused imports of OCAAMS 4) There's a duplicated method createRmContainer in AbstractYarnScheduler and OCAAMS, could we: Move it to SchedulerUtils? And in addition, it will only allocate OpportunisticContainer, so could you rename to createOpportuniscRMContainer. I'm asking this because it cannot guarantee atomic of app and node, I want to make sure no guaranteed container allocation uses this method. 5) RMContainerImpl: Use volatile to avoid synchronized lock for set/getContainer? Because all other methods are using R/W lock.
        Hide
        asuresh Arun Suresh added a comment - - edited

        Updated patch. Do give it another review Wangda Tan

        Can we make scheduler.allocate to have a single ContainerUpdateRequests, which include 4 lists (increase/decrease/promotion/demotion) instead of making all of them become top-level parameter, similarly for Allocation.

        I've created a holder class ContainerUpdateContext.ContainerUpdateRequests which is created in the RMServerUtils#validate.. method and passed around.
        I don't think we need one for the Allocation object, since technically, it itself is a holder for all the lists of allocated/updated containers.

        Agree to all other suggestions, and its incorporated in the patch.

        Show
        asuresh Arun Suresh added a comment - - edited Updated patch. Do give it another review Wangda Tan Can we make scheduler.allocate to have a single ContainerUpdateRequests, which include 4 lists (increase/decrease/promotion/demotion) instead of making all of them become top-level parameter, similarly for Allocation. I've created a holder class ContainerUpdateContext.ContainerUpdateRequests which is created in the RMServerUtils#validate.. method and passed around. I don't think we need one for the Allocation object, since technically, it itself is a holder for all the lists of allocated/updated containers. Agree to all other suggestions, and its incorporated in the patch.
        Hide
        asuresh Arun Suresh added a comment -

        Updating patch v005
        Some minor updates: Moved ContainerUpdateRequests to its own class and removed updateErrors from it.

        Show
        asuresh Arun Suresh added a comment - Updating patch v005 Some minor updates: Moved ContainerUpdateRequests to its own class and removed updateErrors from it.
        Hide
        leftnoteasy Wangda Tan added a comment -

        +1, pending Jenkins, will wait 1 day before commit.

        Arun Suresh, could you update a patch for trunk?

        Show
        leftnoteasy Wangda Tan added a comment - +1, pending Jenkins, will wait 1 day before commit. Arun Suresh , could you update a patch for trunk?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 16 new or modified test files.
        0 mvndep 2m 2s Maven dependency ordering for branch
        +1 mvninstall 13m 6s YARN-5085 passed
        +1 compile 10m 4s YARN-5085 passed
        +1 checkstyle 1m 55s YARN-5085 passed
        +1 mvnsite 2m 59s YARN-5085 passed
        +1 mvneclipse 2m 2s YARN-5085 passed
        +1 findbugs 4m 48s YARN-5085 passed
        +1 javadoc 2m 22s YARN-5085 passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 2m 21s the patch passed
        +1 compile 9m 17s the patch passed
        +1 cc 9m 17s the patch passed
        +1 javac 9m 17s the patch passed
        -0 checkstyle 1m 59s root: The patch generated 37 new + 1695 unchanged - 18 fixed = 1732 total (was 1713)
        +1 mvnsite 3m 18s the patch passed
        +1 mvneclipse 2m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 5m 54s the patch passed
        -1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 912 unchanged - 1 fixed = 913 total (was 913)
        +1 unit 0m 37s hadoop-yarn-api in the patch passed.
        +1 unit 0m 38s hadoop-yarn-server-common in the patch passed.
        +1 unit 39m 0s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 unit 16m 14s hadoop-yarn-client in the patch passed.
        +1 unit 8m 44s hadoop-mapreduce-client-app in the patch passed.
        +1 unit 1m 5s hadoop-sls in the patch passed.
        +1 asflicense 0m 39s The patch does not generate ASF License warnings.
        159m 56s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845664/YARN-5959-YARN-5085.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 3d743a88bb1e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision YARN-5085 / 6938b67
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14557/artifact/patchprocess/diff-checkstyle-root.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14557/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14557/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14557/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 16 new or modified test files. 0 mvndep 2m 2s Maven dependency ordering for branch +1 mvninstall 13m 6s YARN-5085 passed +1 compile 10m 4s YARN-5085 passed +1 checkstyle 1m 55s YARN-5085 passed +1 mvnsite 2m 59s YARN-5085 passed +1 mvneclipse 2m 2s YARN-5085 passed +1 findbugs 4m 48s YARN-5085 passed +1 javadoc 2m 22s YARN-5085 passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 21s the patch passed +1 compile 9m 17s the patch passed +1 cc 9m 17s the patch passed +1 javac 9m 17s the patch passed -0 checkstyle 1m 59s root: The patch generated 37 new + 1695 unchanged - 18 fixed = 1732 total (was 1713) +1 mvnsite 3m 18s the patch passed +1 mvneclipse 2m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 54s the patch passed -1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 912 unchanged - 1 fixed = 913 total (was 913) +1 unit 0m 37s hadoop-yarn-api in the patch passed. +1 unit 0m 38s hadoop-yarn-server-common in the patch passed. +1 unit 39m 0s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 16m 14s hadoop-yarn-client in the patch passed. +1 unit 8m 44s hadoop-mapreduce-client-app in the patch passed. +1 unit 1m 5s hadoop-sls in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 159m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845664/YARN-5959-YARN-5085.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 3d743a88bb1e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-5085 / 6938b67 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14557/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14557/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14557/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14557/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Attaching trunk patch..
        Thanks for the review Wangda Tan.

        Show
        asuresh Arun Suresh added a comment - Attaching trunk patch.. Thanks for the review Wangda Tan .
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 16 new or modified test files.
        0 mvndep 1m 48s Maven dependency ordering for branch
        +1 mvninstall 14m 30s trunk passed
        +1 compile 11m 15s trunk passed
        +1 checkstyle 2m 2s trunk passed
        +1 mvnsite 3m 20s trunk passed
        +1 mvneclipse 2m 14s trunk passed
        +1 findbugs 5m 32s trunk passed
        +1 javadoc 2m 33s trunk passed
        0 mvndep 0m 18s Maven dependency ordering for patch
        +1 mvninstall 2m 48s the patch passed
        +1 compile 10m 43s the patch passed
        +1 cc 10m 43s the patch passed
        +1 javac 10m 43s the patch passed
        -0 checkstyle 2m 7s root: The patch generated 37 new + 1694 unchanged - 18 fixed = 1731 total (was 1712)
        +1 mvnsite 3m 41s the patch passed
        +1 mvneclipse 2m 29s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 6m 38s the patch passed
        +1 javadoc 0m 30s hadoop-yarn-api in the patch passed.
        +1 javadoc 0m 28s hadoop-yarn-server-common in the patch passed.
        +1 javadoc 0m 34s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 912 unchanged - 1 fixed = 912 total (was 913)
        +1 javadoc 0m 27s hadoop-yarn-client in the patch passed.
        +1 javadoc 0m 28s hadoop-mapreduce-client-app in the patch passed.
        +1 javadoc 0m 26s hadoop-sls in the patch passed.
        +1 unit 0m 40s hadoop-yarn-api in the patch passed.
        +1 unit 0m 41s hadoop-yarn-server-common in the patch passed.
        +1 unit 41m 23s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 unit 16m 42s hadoop-yarn-client in the patch passed.
        +1 unit 9m 34s hadoop-mapreduce-client-app in the patch passed.
        +1 unit 1m 11s hadoop-sls in the patch passed.
        +1 asflicense 0m 41s The patch does not generate ASF License warnings.
        171m 33s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845716/YARN-5959.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux d508484ab892 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a605ff3
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14560/artifact/patchprocess/diff-checkstyle-root.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14560/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14560/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 16 new or modified test files. 0 mvndep 1m 48s Maven dependency ordering for branch +1 mvninstall 14m 30s trunk passed +1 compile 11m 15s trunk passed +1 checkstyle 2m 2s trunk passed +1 mvnsite 3m 20s trunk passed +1 mvneclipse 2m 14s trunk passed +1 findbugs 5m 32s trunk passed +1 javadoc 2m 33s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 48s the patch passed +1 compile 10m 43s the patch passed +1 cc 10m 43s the patch passed +1 javac 10m 43s the patch passed -0 checkstyle 2m 7s root: The patch generated 37 new + 1694 unchanged - 18 fixed = 1731 total (was 1712) +1 mvnsite 3m 41s the patch passed +1 mvneclipse 2m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 38s the patch passed +1 javadoc 0m 30s hadoop-yarn-api in the patch passed. +1 javadoc 0m 28s hadoop-yarn-server-common in the patch passed. +1 javadoc 0m 34s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 912 unchanged - 1 fixed = 912 total (was 913) +1 javadoc 0m 27s hadoop-yarn-client in the patch passed. +1 javadoc 0m 28s hadoop-mapreduce-client-app in the patch passed. +1 javadoc 0m 26s hadoop-sls in the patch passed. +1 unit 0m 40s hadoop-yarn-api in the patch passed. +1 unit 0m 41s hadoop-yarn-server-common in the patch passed. +1 unit 41m 23s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 16m 42s hadoop-yarn-client in the patch passed. +1 unit 9m 34s hadoop-mapreduce-client-app in the patch passed. +1 unit 1m 11s hadoop-sls in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 171m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845716/YARN-5959.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux d508484ab892 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a605ff3 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14560/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14560/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14560/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committing ...

        Show
        leftnoteasy Wangda Tan added a comment - Committing ...
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to trunk, thanks Arun Suresh.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to trunk, thanks Arun Suresh .
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11075 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11075/)
        YARN-5959. RM changes to support change of container ExecutionType. (wangda: rev 0a55bd841ec0f2eb89a0383f4c589526e8b138d4)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/UpdateContainerRequest.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Allocation.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoAppAttempt.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java
        • (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
        • (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/OpportunisticContainerContext.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerUpdateType.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/UpdateContainerError.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/SchedulerRequestKey.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SchedulingPlacementSet.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerUpdateContext.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/OpportunisticContainerAllocator.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerUpdates.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11075 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11075/ ) YARN-5959 . RM changes to support change of container ExecutionType. (wangda: rev 0a55bd841ec0f2eb89a0383f4c589526e8b138d4) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/UpdateContainerRequest.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Allocation.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoAppAttempt.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/OpportunisticContainerContext.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerUpdateType.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/UpdateContainerError.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/SchedulerRequestKey.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SchedulingPlacementSet.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerUpdateContext.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/scheduler/OpportunisticContainerAllocator.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerUpdates.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
        Hide
        asuresh Arun Suresh added a comment -

        Re-openeing to include patch for branch-2

        Show
        asuresh Arun Suresh added a comment - Re-openeing to include patch for branch-2
        Hide
        asuresh Arun Suresh added a comment -

        Attaching patch for branch-2

        Show
        asuresh Arun Suresh added a comment - Attaching patch for branch-2
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 16 new or modified test files.
        0 mvndep 1m 39s Maven dependency ordering for branch
        +1 mvninstall 7m 54s branch-2 passed
        +1 compile 7m 0s branch-2 passed with JDK v1.8.0_121
        +1 compile 6m 56s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 1m 47s branch-2 passed
        +1 mvnsite 2m 53s branch-2 passed
        +1 mvneclipse 1m 29s branch-2 passed
        -1 findbugs 0m 17s hadoop-mapreduce-client-app in branch-2 failed.
        +1 javadoc 1m 47s branch-2 passed with JDK v1.8.0_121
        +1 javadoc 2m 0s branch-2 passed with JDK v1.7.0_121
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 2m 28s the patch passed
        +1 compile 6m 45s the patch passed with JDK v1.8.0_121
        +1 cc 6m 45s the patch passed
        +1 javac 6m 45s the patch passed
        +1 compile 6m 52s the patch passed with JDK v1.7.0_121
        +1 cc 6m 52s the patch passed
        +1 javac 6m 52s the patch passed
        -0 checkstyle 1m 55s root: The patch generated 38 new + 1600 unchanged - 18 fixed = 1638 total (was 1618)
        +1 mvnsite 3m 9s the patch passed
        +1 mvneclipse 1m 48s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 6m 53s the patch passed
        -1 javadoc 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 907 unchanged - 1 fixed = 908 total (was 908)
        +1 javadoc 2m 27s the patch passed with JDK v1.7.0_121
        +1 unit 0m 33s hadoop-yarn-api in the patch passed with JDK v1.7.0_121.
        +1 unit 0m 34s hadoop-yarn-server-common in the patch passed with JDK v1.7.0_121.
        -1 unit 41m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        +1 unit 17m 0s hadoop-yarn-client in the patch passed with JDK v1.7.0_121.
        +1 unit 8m 59s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_121.
        -1 unit 0m 42s hadoop-sls in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        236m 19s



        Reason Tests
        JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.sls.TestSLSRunner
        JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.sls.TestSLSRunner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854251/YARN-5959-branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux e7349346a5f0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 8013577
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/diff-checkstyle-root.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-sls-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15062/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15062/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 16 new or modified test files. 0 mvndep 1m 39s Maven dependency ordering for branch +1 mvninstall 7m 54s branch-2 passed +1 compile 7m 0s branch-2 passed with JDK v1.8.0_121 +1 compile 6m 56s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 1m 47s branch-2 passed +1 mvnsite 2m 53s branch-2 passed +1 mvneclipse 1m 29s branch-2 passed -1 findbugs 0m 17s hadoop-mapreduce-client-app in branch-2 failed. +1 javadoc 1m 47s branch-2 passed with JDK v1.8.0_121 +1 javadoc 2m 0s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 28s the patch passed +1 compile 6m 45s the patch passed with JDK v1.8.0_121 +1 cc 6m 45s the patch passed +1 javac 6m 45s the patch passed +1 compile 6m 52s the patch passed with JDK v1.7.0_121 +1 cc 6m 52s the patch passed +1 javac 6m 52s the patch passed -0 checkstyle 1m 55s root: The patch generated 38 new + 1600 unchanged - 18 fixed = 1638 total (was 1618) +1 mvnsite 3m 9s the patch passed +1 mvneclipse 1m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 53s the patch passed -1 javadoc 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 907 unchanged - 1 fixed = 908 total (was 908) +1 javadoc 2m 27s the patch passed with JDK v1.7.0_121 +1 unit 0m 33s hadoop-yarn-api in the patch passed with JDK v1.7.0_121. +1 unit 0m 34s hadoop-yarn-server-common in the patch passed with JDK v1.7.0_121. -1 unit 41m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 unit 17m 0s hadoop-yarn-client in the patch passed with JDK v1.7.0_121. +1 unit 8m 59s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_121. -1 unit 0m 42s hadoop-sls in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 236m 19s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.sls.TestSLSRunner JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.sls.TestSLSRunner Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854251/YARN-5959-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux e7349346a5f0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 8013577 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15062/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-sls-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15062/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15062/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 27s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 16 new or modified test files.
        0 mvndep 1m 9s Maven dependency ordering for branch
        +1 mvninstall 6m 56s branch-2 passed
        +1 compile 6m 49s branch-2 passed with JDK v1.8.0_121
        +1 compile 6m 47s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 1m 45s branch-2 passed
        +1 mvnsite 2m 51s branch-2 passed
        +1 mvneclipse 1m 30s branch-2 passed
        +1 findbugs 5m 26s branch-2 passed
        +1 javadoc 1m 45s branch-2 passed with JDK v1.8.0_121
        +1 javadoc 1m 59s branch-2 passed with JDK v1.7.0_121
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 2m 31s the patch passed
        +1 compile 6m 37s the patch passed with JDK v1.8.0_121
        +1 cc 6m 37s the patch passed
        +1 javac 6m 37s the patch passed
        +1 compile 6m 49s the patch passed with JDK v1.7.0_121
        +1 cc 6m 49s the patch passed
        +1 javac 6m 49s the patch passed
        -0 checkstyle 1m 54s root: The patch generated 38 new + 1600 unchanged - 18 fixed = 1638 total (was 1618)
        +1 mvnsite 3m 10s the patch passed
        +1 mvneclipse 1m 49s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 6m 56s the patch passed
        -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 907 unchanged - 1 fixed = 908 total (was 908)
        +1 javadoc 2m 22s the patch passed with JDK v1.7.0_121
        +1 unit 0m 32s hadoop-yarn-api in the patch passed with JDK v1.7.0_121.
        +1 unit 0m 34s hadoop-yarn-server-common in the patch passed with JDK v1.7.0_121.
        -1 unit 41m 5s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        +1 unit 18m 0s hadoop-yarn-client in the patch passed with JDK v1.7.0_121.
        +1 unit 9m 13s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_121.
        -1 unit 0m 42s hadoop-sls in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        236m 47s



        Reason Tests
        JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
          hadoop.yarn.sls.TestSLSRunner
        JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.sls.TestSLSRunner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854251/YARN-5959-branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 6fc7197341aa 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 8013577
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15063/artifact/patchprocess/diff-checkstyle-root.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15063/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15063/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15063/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-sls-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15063/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15063/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 16 new or modified test files. 0 mvndep 1m 9s Maven dependency ordering for branch +1 mvninstall 6m 56s branch-2 passed +1 compile 6m 49s branch-2 passed with JDK v1.8.0_121 +1 compile 6m 47s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 1m 45s branch-2 passed +1 mvnsite 2m 51s branch-2 passed +1 mvneclipse 1m 30s branch-2 passed +1 findbugs 5m 26s branch-2 passed +1 javadoc 1m 45s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 59s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 31s the patch passed +1 compile 6m 37s the patch passed with JDK v1.8.0_121 +1 cc 6m 37s the patch passed +1 javac 6m 37s the patch passed +1 compile 6m 49s the patch passed with JDK v1.7.0_121 +1 cc 6m 49s the patch passed +1 javac 6m 49s the patch passed -0 checkstyle 1m 54s root: The patch generated 38 new + 1600 unchanged - 18 fixed = 1638 total (was 1618) +1 mvnsite 3m 10s the patch passed +1 mvneclipse 1m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 56s the patch passed -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121 with JDK v1.8.0_121 generated 1 new + 907 unchanged - 1 fixed = 908 total (was 908) +1 javadoc 2m 22s the patch passed with JDK v1.7.0_121 +1 unit 0m 32s hadoop-yarn-api in the patch passed with JDK v1.7.0_121. +1 unit 0m 34s hadoop-yarn-server-common in the patch passed with JDK v1.7.0_121. -1 unit 41m 5s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 unit 18m 0s hadoop-yarn-client in the patch passed with JDK v1.7.0_121. +1 unit 9m 13s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_121. -1 unit 0m 42s hadoop-sls in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 236m 47s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   hadoop.yarn.sls.TestSLSRunner JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.sls.TestSLSRunner Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854251/YARN-5959-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 6fc7197341aa 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 8013577 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15063/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15063/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15063/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15063/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-sls-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15063/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15063/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Fixing javadoc warning, some checkstyles and the SLS testcase. The remaining test failures are not related and the TestCapacityScheduler works fine for me.

        Show
        asuresh Arun Suresh added a comment - Fixing javadoc warning, some checkstyles and the SLS testcase. The remaining test failures are not related and the TestCapacityScheduler works fine for me.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 33s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 16 new or modified test files.
        0 mvndep 1m 15s Maven dependency ordering for branch
        +1 mvninstall 6m 57s branch-2 passed
        +1 compile 6m 19s branch-2 passed with JDK v1.8.0_121
        +1 compile 6m 47s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 1m 48s branch-2 passed
        +1 mvnsite 2m 50s branch-2 passed
        +1 mvneclipse 1m 37s branch-2 passed
        +1 findbugs 5m 27s branch-2 passed
        +1 javadoc 2m 36s branch-2 passed with JDK v1.8.0_121
        +1 javadoc 2m 44s branch-2 passed with JDK v1.7.0_121
        0 mvndep 0m 22s Maven dependency ordering for patch
        +1 mvninstall 3m 6s the patch passed
        +1 compile 6m 19s the patch passed with JDK v1.8.0_121
        +1 cc 6m 19s the patch passed
        +1 javac 6m 19s the patch passed
        +1 compile 7m 51s the patch passed with JDK v1.7.0_121
        +1 cc 7m 51s the patch passed
        +1 javac 7m 51s the patch passed
        -0 checkstyle 2m 5s root: The patch generated 34 new + 1600 unchanged - 18 fixed = 1634 total (was 1618)
        +1 mvnsite 3m 19s the patch passed
        +1 mvneclipse 1m 52s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 6m 48s the patch passed
        +1 javadoc 0m 23s hadoop-yarn-api in the patch passed with JDK v1.8.0_121.
        +1 javadoc 0m 20s hadoop-yarn-server-common in the patch passed with JDK v1.8.0_121.
        +1 javadoc 0m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121 with JDK v1.8.0_121 generated 0 new + 907 unchanged - 1 fixed = 907 total (was 908)
        +1 javadoc 0m 18s hadoop-yarn-client in the patch passed with JDK v1.8.0_121.
        +1 javadoc 0m 20s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_121.
        +1 javadoc 0m 18s hadoop-sls in the patch passed with JDK v1.8.0_121.
        +1 javadoc 2m 16s the patch passed with JDK v1.7.0_121
        +1 unit 0m 33s hadoop-yarn-api in the patch passed with JDK v1.7.0_121.
        +1 unit 0m 34s hadoop-yarn-server-common in the patch passed with JDK v1.7.0_121.
        -1 unit 40m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        -1 unit 20m 6s hadoop-yarn-client in the patch failed with JDK v1.7.0_121.
        +1 unit 9m 18s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_121.
        +1 unit 1m 7s hadoop-sls in the patch passed with JDK v1.7.0_121.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        241m 47s



        Reason Tests
        JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing
        JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.client.api.impl.TestAMRMProxy
          hadoop.yarn.client.TestApplicationClientProtocolOnHA



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue YARN-5959
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854348/YARN-5959-branch-2.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 74ba55ce3a8d 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 40bc9e7
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15071/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15071/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15071/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15071/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15071/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 16 new or modified test files. 0 mvndep 1m 15s Maven dependency ordering for branch +1 mvninstall 6m 57s branch-2 passed +1 compile 6m 19s branch-2 passed with JDK v1.8.0_121 +1 compile 6m 47s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 1m 48s branch-2 passed +1 mvnsite 2m 50s branch-2 passed +1 mvneclipse 1m 37s branch-2 passed +1 findbugs 5m 27s branch-2 passed +1 javadoc 2m 36s branch-2 passed with JDK v1.8.0_121 +1 javadoc 2m 44s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 3m 6s the patch passed +1 compile 6m 19s the patch passed with JDK v1.8.0_121 +1 cc 6m 19s the patch passed +1 javac 6m 19s the patch passed +1 compile 7m 51s the patch passed with JDK v1.7.0_121 +1 cc 7m 51s the patch passed +1 javac 7m 51s the patch passed -0 checkstyle 2m 5s root: The patch generated 34 new + 1600 unchanged - 18 fixed = 1634 total (was 1618) +1 mvnsite 3m 19s the patch passed +1 mvneclipse 1m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 48s the patch passed +1 javadoc 0m 23s hadoop-yarn-api in the patch passed with JDK v1.8.0_121. +1 javadoc 0m 20s hadoop-yarn-server-common in the patch passed with JDK v1.8.0_121. +1 javadoc 0m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_121 with JDK v1.8.0_121 generated 0 new + 907 unchanged - 1 fixed = 907 total (was 908) +1 javadoc 0m 18s hadoop-yarn-client in the patch passed with JDK v1.8.0_121. +1 javadoc 0m 20s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_121. +1 javadoc 0m 18s hadoop-sls in the patch passed with JDK v1.8.0_121. +1 javadoc 2m 16s the patch passed with JDK v1.7.0_121 +1 unit 0m 33s hadoop-yarn-api in the patch passed with JDK v1.7.0_121. +1 unit 0m 34s hadoop-yarn-server-common in the patch passed with JDK v1.7.0_121. -1 unit 40m 49s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. -1 unit 20m 6s hadoop-yarn-client in the patch failed with JDK v1.7.0_121. +1 unit 9m 18s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_121. +1 unit 1m 7s hadoop-sls in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 241m 47s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.TestApplicationClientProtocolOnHA Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-5959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854348/YARN-5959-branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 74ba55ce3a8d 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 40bc9e7 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15071/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15071/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15071/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15071/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15071/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        The testcase failures are un-related. Filed YARN-6231 to track the TestFairScheduler failure.
        Will be checking this in shortly

        Show
        asuresh Arun Suresh added a comment - The testcase failures are un-related. Filed YARN-6231 to track the TestFairScheduler failure. Will be checking this in shortly
        Hide
        asuresh Arun Suresh added a comment -

        Committed this to branch-2 as well

        Show
        asuresh Arun Suresh added a comment - Committed this to branch-2 as well

          People

          • Assignee:
            asuresh Arun Suresh
            Reporter:
            asuresh Arun Suresh
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development