Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1197 Support changing resources of an allocated container
  3. YARN-1644

RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    1. yarn-1644.1.patch
      17 kB
      Wangda Tan
    2. YARN-1644.1.patch
      19 kB
      MENG DING
    3. YARN-1644.2.patch
      20 kB
      MENG DING
    4. YARN-1644.3.patch
      24 kB
      MENG DING
    5. YARN-1644-YARN-1197.4.patch
      27 kB
      MENG DING
    6. YARN-1644-YARN-1197.5.patch
      27 kB
      MENG DING
    7. YARN-1644-YARN-1197.6.patch
      51 kB
      MENG DING

      Issue Links

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        Added patch#1 for review

        Show
        leftnoteasy Wangda Tan added a comment - Added patch#1 for review
        Hide
        leftnoteasy Wangda Tan added a comment -

        This should be included by YARN-1449.

        Show
        leftnoteasy Wangda Tan added a comment - This should be included by YARN-1449 .
        Hide
        mding MENG DING added a comment -

        Reopen this issue to track RM-NM protocol and NodeStatusUpdater implementation

        Show
        mding MENG DING added a comment - Reopen this issue to track RM-NM protocol and NodeStatusUpdater implementation
        Hide
        mding MENG DING added a comment -

        Attach patch for RM-NM protocol changes and NodeStatusUpdater Implementation.

        Show
        mding MENG DING added a comment - Attach patch for RM-NM protocol changes and NodeStatusUpdater Implementation.
        Hide
        mding MENG DING added a comment -

        Fixed coding style issues, and added test case to TestYarnServerApiClasses

        Show
        mding MENG DING added a comment - Fixed coding style issues, and added test case to TestYarnServerApiClasses
        Hide
        mding MENG DING added a comment -

        Updated this patch as dependent patch has been updated.

        Show
        mding MENG DING added a comment - Updated this patch as dependent patch has been updated.
        Hide
        mding MENG DING added a comment -

        Attach latest patch and trigger jenkins.

        This patch does the following:

        • Adds RM-NM protocol changes and PB implementations
        • Updates NodeStatusUpdater logic to support container resizing. i.e., get decreased containers from heartbeat response and send out the CMgrDecreaseContainersResourceEvent to ContainerManagerImpl; gets the increased containers from NMContext and set it in heartbeat request to send to RM
        • Introduces the increasedContainers map in NMContext to keep track of containers whose resource are being increased.
        Show
        mding MENG DING added a comment - Attach latest patch and trigger jenkins. This patch does the following: Adds RM-NM protocol changes and PB implementations Updates NodeStatusUpdater logic to support container resizing. i.e., get decreased containers from heartbeat response and send out the CMgrDecreaseContainersResourceEvent to ContainerManagerImpl ; gets the increased containers from NMContext and set it in heartbeat request to send to RM Introduces the increasedContainers map in NMContext to keep track of containers whose resource are being increased.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 29s Findbugs (version ) appears to be broken on YARN-1197.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 30s There were no new javac warning messages.
        +1 javadoc 10m 26s There were no new javadoc warning messages.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 18s The applied patch generated 2 new checkstyle issues (total was 47, now 49).
        +1 whitespace 0m 5s The patch has no lines that end in whitespace.
        +1 install 1m 26s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 3m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 6m 57s Tests passed in hadoop-yarn-client.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
        -1 yarn tests 6m 18s Tests failed in hadoop-yarn-server-nodemanager.
            58m 31s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12746561/YARN-1644-YARN-1197.5.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision YARN-1197 / 2934ff3
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/YARN-1197FindbugsWarningshadoop-yarn-server-common.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
        hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/testrun_hadoop-yarn-client.txt
        hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8614/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8614/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 29s Findbugs (version ) appears to be broken on YARN-1197 . +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 30s There were no new javac warning messages. +1 javadoc 10m 26s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 18s The applied patch generated 2 new checkstyle issues (total was 47, now 49). +1 whitespace 0m 5s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 3m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 6m 57s Tests passed in hadoop-yarn-client. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. -1 yarn tests 6m 18s Tests failed in hadoop-yarn-server-nodemanager.     58m 31s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12746561/YARN-1644-YARN-1197.5.patch Optional Tests javac unit findbugs checkstyle javadoc git revision YARN-1197 / 2934ff3 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/YARN-1197FindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8614/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8614/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8614/console This message was automatically generated.
        Hide
        mding MENG DING added a comment -
        • The checkstyle reports are false alarms:
          ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java:440:29: 'containersToDecrease' hides a field.
          ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java:371:23: 'increasedContainers' hides a field.
          
        • The failed test passed in my local environment, so it is probably unrelated.
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.954 sec - in org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels
          
          Results :
          
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
          
        Show
        mding MENG DING added a comment - The checkstyle reports are false alarms: ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java:440:29: 'containersToDecrease' hides a field. ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java:371:23: 'increasedContainers' hides a field. The failed test passed in my local environment, so it is probably unrelated. ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.954 sec - in org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
        Hide
        jianhe Jian He added a comment -

        thanks MENG DING! one moment on the patch:

        • NodeStatusUpdaterImpl#getIncreasedContainers and the change in ContainerManagerImpl:
          If RM restarts right after NM sends the increased containers, NM may not yet update the container status and still sends the old resource and therefore RM will recover the container with old resource. How about we re-use the same containerStatus object for the increasedContainer ? this way we can also avoid this situation.
        Show
        jianhe Jian He added a comment - thanks MENG DING ! one moment on the patch: NodeStatusUpdaterImpl#getIncreasedContainers and the change in ContainerManagerImpl: If RM restarts right after NM sends the increased containers, NM may not yet update the container status and still sends the old resource and therefore RM will recover the container with old resource. How about we re-use the same containerStatus object for the increasedContainer ? this way we can also avoid this situation.
        Hide
        mding MENG DING added a comment -

        Thanks Jian He for the review comments, and thanks for bringing up the corner case that I have missed.

        I think the main problem (if I understand it correctly) right now is that if NM fails to send the increasedContainers to RM, and gets a Resync instruction (due to RM restart), the increasedContainers is still cleared from NMContext. If at this moment, the container status has not yet been updated in NM (its a very short time window, but still possible), the registerNodeManager will send old container resource info to RM for RM container recovery.

        If this is the case, then reusing ContainerStatus object for increasedContainers still cannot resolve the problem. What we need, I believe, is to make sure that we only remove the container from NMContext.increasedContainers after the container status has been updated in NM. We also need to add increasedContainers to the RegisterNodeManagerRequestProto, such that during node manager registration, RM needs to check RegisterNodeManagerRequest.increasedContainers as well to set the correct container size for container recovery.

        Thoughts?

        Show
        mding MENG DING added a comment - Thanks Jian He for the review comments, and thanks for bringing up the corner case that I have missed. I think the main problem (if I understand it correctly) right now is that if NM fails to send the increasedContainers to RM, and gets a Resync instruction (due to RM restart), the increasedContainers is still cleared from NMContext . If at this moment, the container status has not yet been updated in NM (its a very short time window, but still possible), the registerNodeManager will send old container resource info to RM for RM container recovery. If this is the case, then reusing ContainerStatus object for increasedContainers still cannot resolve the problem. What we need, I believe, is to make sure that we only remove the container from NMContext.increasedContainers after the container status has been updated in NM. We also need to add increasedContainers to the RegisterNodeManagerRequestProto , such that during node manager registration, RM needs to check RegisterNodeManagerRequest.increasedContainers as well to set the correct container size for container recovery. Thoughts?
        Hide
        jianhe Jian He added a comment - - edited

        if NM fails to send the increasedContainers to RM,

        Even if NM succeeded to send the increasedContainers to RM, if NM re-registers back with RM before the container size is updated, RM will also recover the container with old resources.

        The solution I have in mind is that, we do not keep track of extra increasedContainers in NMContext. We always rely on NMContext#containers to send the container status. RM will check container size based on the containerStatus in node heartbeat.

        Show
        jianhe Jian He added a comment - - edited if NM fails to send the increasedContainers to RM, Even if NM succeeded to send the increasedContainers to RM, if NM re-registers back with RM before the container size is updated, RM will also recover the container with old resources. The solution I have in mind is that, we do not keep track of extra increasedContainers in NMContext. We always rely on NMContext#containers to send the container status. RM will check container size based on the containerStatus in node heartbeat.
        Hide
        mding MENG DING added a comment -

        Even if NM succeeded to send the increasedContainers to RM, if NM re-registers back with RM before the container size is updated, RM will also recover the container with old resources.

        The increasedContainers contains the target resource size for each container being increased, so if NM-registers back with RM before the container size is updated, RM will check both containerStatus (which contains the old resource for the container) and increasedContainers (which contains the target resource for the container) in the RegisterNodeManager request, and will be able to recover the correct container size.

        The solution I have in mind is that, we do not keep track of extra increasedContainers in NMContext. We always rely on NMContext#containers to send the container status. RM will check container size based on the containerStatus in node heartbeat.

        The question I have with this solution is: how does RM know that an increase has been successfully completed in NM without an explicit protocol? Does RM keep checking the size of each container reported by NM from heartbeat to heartbeat, and decide that an increase has been completed if the container size from the previous heartbeat is smaller than the container size from the current heartbeat? I think this won't work in the RM restart scenario you mentioned. Consider the following sequence of events:

        • RM restarts while there is an increase going on in NM
        • NM re-registers with RM before the container size is updated in NM, and RM recovers all containers with old resources, and builds up its internal resource bookkeeping for scheduler
        • Later on container size is updated in NM, and RM gets the increased container size in the next heartbeat request. What should RM do now? It cannot simply go ahead to increase the resource bookkeeping in its scheduler, because the scheduler did not allocate the extra resource after restart.

        IMHO, it is crucial for RM to recover the correct container size during the NM registration if there is a pending container resource increase action going on in NM, that is the reason I propose to add the increasedContainers to the RegisterNodeManagerRequestProto, and also make sure that a container is only removed from increasedContainers when its resize is completed in NM.

        Thoughts/comments?

        Show
        mding MENG DING added a comment - Even if NM succeeded to send the increasedContainers to RM, if NM re-registers back with RM before the container size is updated, RM will also recover the container with old resources. The increasedContainers contains the target resource size for each container being increased, so if NM-registers back with RM before the container size is updated, RM will check both containerStatus (which contains the old resource for the container) and increasedContainers (which contains the target resource for the container) in the RegisterNodeManager request, and will be able to recover the correct container size. The solution I have in mind is that, we do not keep track of extra increasedContainers in NMContext. We always rely on NMContext#containers to send the container status. RM will check container size based on the containerStatus in node heartbeat. The question I have with this solution is: how does RM know that an increase has been successfully completed in NM without an explicit protocol? Does RM keep checking the size of each container reported by NM from heartbeat to heartbeat, and decide that an increase has been completed if the container size from the previous heartbeat is smaller than the container size from the current heartbeat? I think this won't work in the RM restart scenario you mentioned. Consider the following sequence of events: RM restarts while there is an increase going on in NM NM re-registers with RM before the container size is updated in NM, and RM recovers all containers with old resources, and builds up its internal resource bookkeeping for scheduler Later on container size is updated in NM, and RM gets the increased container size in the next heartbeat request. What should RM do now? It cannot simply go ahead to increase the resource bookkeeping in its scheduler, because the scheduler did not allocate the extra resource after restart. IMHO, it is crucial for RM to recover the correct container size during the NM registration if there is a pending container resource increase action going on in NM, that is the reason I propose to add the increasedContainers to the RegisterNodeManagerRequestProto , and also make sure that a container is only removed from increasedContainers when its resize is completed in NM. Thoughts/comments?
        Hide
        jianhe Jian He added a comment -

        actually, if we change YARN-3867 to update the resource synchronously. Then, this race condition can be mostly avoided and RM can rely on the container status to recover the correct resource ?

        Show
        jianhe Jian He added a comment - actually, if we change YARN-3867 to update the resource synchronously. Then, this race condition can be mostly avoided and RM can rely on the container status to recover the correct resource ?
        Hide
        mding MENG DING added a comment -

        I think the race condition is still there. Even though the resource is updated synchronously, NM re-registration can still happen between the time the increase action is accepted, and the time the resource is updated. We still need a mechanism for RM to know that there is pending increase going on in NM, don't we?

        Show
        mding MENG DING added a comment - I think the race condition is still there. Even though the resource is updated synchronously, NM re-registration can still happen between the time the increase action is accepted, and the time the resource is updated. We still need a mechanism for RM to know that there is pending increase going on in NM, don't we?
        Hide
        jianhe Jian He added a comment -

        add the increasedContainers to the RegisterNodeManagerRequestProto, make sure that a container is only removed from increasedContainers when its resize is completed in NM.

        IIUC, this has the same problem? NM re-registration can still happen between the time the increase action is accepted, and the time it's added into increasedContainers. Even startContainer has the same problem, newly started container may fall into this tiny window that RM won't recover this container. Maybe RM could react on the node heartbeat with respect to increase container as necessary too to handle this race condition ?

        Show
        jianhe Jian He added a comment - add the increasedContainers to the RegisterNodeManagerRequestProto, make sure that a container is only removed from increasedContainers when its resize is completed in NM. IIUC, this has the same problem? NM re-registration can still happen between the time the increase action is accepted, and the time it's added into increasedContainers. Even startContainer has the same problem, newly started container may fall into this tiny window that RM won't recover this container. Maybe RM could react on the node heartbeat with respect to increase container as necessary too to handle this race condition ?
        Hide
        mding MENG DING added a comment -

        NM re-registration can still happen between the time the increase action is accepted, and the time it's added into increasedContainers. Even startContainer has the same problem, newly started container may fall into this tiny window that RM won't recover this container.

        Yes, you are right that startContainer would have the same problem.
        So to make it clear, RM restart/NM re-registration can happen in the following scenarios:

        • 1. Container resource increase is already completed. In this case, NM re-registration can send the correct (increased) container size (through containerStatus object) for RM recovery.
        • 2. Container to be increased has been added into increasedContainers, but the resource is not yet updated. In this case, NM re-registration can send the correct container size through both containerStatus and increasedContainers objects for RM recovery.
        • 3. The increase action is accepted, but the container to be increased has not been added into increasedContainers. In this case, the resource view between NM and RM becomes different. The same issue applies to startContainers.

        I don't have a solution for c yet, but I think the chance for scenario 3 to happen is very small, especially with the blockNewContainerRequests and matching RM identifier logic right now. Maybe we can log a separate JIRA for scenario 3, and fix that for both container increase and container launch?

        Show
        mding MENG DING added a comment - NM re-registration can still happen between the time the increase action is accepted, and the time it's added into increasedContainers. Even startContainer has the same problem, newly started container may fall into this tiny window that RM won't recover this container. Yes, you are right that startContainer would have the same problem. So to make it clear, RM restart/NM re-registration can happen in the following scenarios: 1. Container resource increase is already completed. In this case, NM re-registration can send the correct (increased) container size (through containerStatus object) for RM recovery. 2. Container to be increased has been added into increasedContainers, but the resource is not yet updated. In this case, NM re-registration can send the correct container size through both containerStatus and increasedContainers objects for RM recovery. 3. The increase action is accepted, but the container to be increased has not been added into increasedContainers. In this case, the resource view between NM and RM becomes different. The same issue applies to startContainers. I don't have a solution for c yet, but I think the chance for scenario 3 to happen is very small, especially with the blockNewContainerRequests and matching RM identifier logic right now. Maybe we can log a separate JIRA for scenario 3, and fix that for both container increase and container launch?
        Hide
        mding MENG DING added a comment -

        I had an offline discussion with Jian He a while ago, and we thought that the race condition in scenario 3 can be handled in a separate JIRA, as it applies to both increase container size and start container.

        For this ticket, we are exploring the idea of getting rid of the increasedContainers list from NM. The increasedContainers was originally introduced as a way to let NM inform RM that an increase action has been completed in NM. However, it seems that we may achieve the same result by checking containerStatuses. In particular, RM will keep checking the difference of container sizes between heartbeats. For each container:

        • If the container size reported from this heartbeat is larger than the size reported from previous heartbeat:
          • If the reported size is the same as RM's bookkeeping for this container, then this is a confirmation of container resource increase.
          • If the reported size is larger than RM's bookkeeping for this container, then this is due to an RM recovery during container resource increase in NM. RM should increase its bookkeeping of this container to match the reported size.
          • If the reported size is smaller than RM's bookkeeping for this container, it should be an error.
        • If the container size reported from this heartbeat is smaller than the size reported from previous heartbeat:
          • If the reported size is the same as RM's bookkeeping for this container, then this is a confirmation of container resource decrease.
          • Any other case should be an error.

        The validity of this approach is still being decided. Any comments/concerns are welcome.

        Show
        mding MENG DING added a comment - I had an offline discussion with Jian He a while ago, and we thought that the race condition in scenario 3 can be handled in a separate JIRA, as it applies to both increase container size and start container. For this ticket, we are exploring the idea of getting rid of the increasedContainers list from NM. The increasedContainers was originally introduced as a way to let NM inform RM that an increase action has been completed in NM. However, it seems that we may achieve the same result by checking containerStatuses . In particular, RM will keep checking the difference of container sizes between heartbeats. For each container: If the container size reported from this heartbeat is larger than the size reported from previous heartbeat: If the reported size is the same as RM's bookkeeping for this container, then this is a confirmation of container resource increase. If the reported size is larger than RM's bookkeeping for this container, then this is due to an RM recovery during container resource increase in NM. RM should increase its bookkeeping of this container to match the reported size. If the reported size is smaller than RM's bookkeeping for this container, it should be an error. If the container size reported from this heartbeat is smaller than the size reported from previous heartbeat: If the reported size is the same as RM's bookkeeping for this container, then this is a confirmation of container resource decrease. Any other case should be an error. The validity of this approach is still being decided. Any comments/concerns are welcome.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Discussed with Jian He, some thoughts:

        There're 3 corner cases we need to handle:
        1. AM send decrease container to RM before send increase container to NM
        2. RM crashes after issued increase container, and AM increase container to NM during NM registering
        3. Same as 2. but AM send decrease container request to RM before RM receives NM reported increase container.

        What we may need to consider is "version of container", RM will add 1 to container version if increased/decreased a container. And container-version will be added to ContainerTokenIdentifier, NM reported increased container and NMContainerStatus while registering.

        From RM's view, it should keep the latest updated container resource. So for above corner cases:
        1. Result: container decreased
        2. Result: container increased
        3. Result: container decreased (because the latest resource AM sent to RM is decrese).

        So in RM side, it will check:

        if (rm.version >= nm.version) {
        	// keep existing container in RM unchanged, and tell NM about this
        	// why include "==" here is, if rm.version == nm.version, corner case #3 happened.
        } else {
        	// change container in RM
        }
        

        So in summary what we need in protocol is:

        • Container-version in ContainerTokenIdentifier
        • COntainer-version in NMContainerStatus
        • add a IncreasedContainer of NM-RM heartbeat, and include container-version in IncreasedContainer.

        Thoughts? MENG DING

        Show
        leftnoteasy Wangda Tan added a comment - Discussed with Jian He , some thoughts: There're 3 corner cases we need to handle: 1. AM send decrease container to RM before send increase container to NM 2. RM crashes after issued increase container, and AM increase container to NM during NM registering 3. Same as 2. but AM send decrease container request to RM before RM receives NM reported increase container. What we may need to consider is "version of container", RM will add 1 to container version if increased/decreased a container. And container-version will be added to ContainerTokenIdentifier, NM reported increased container and NMContainerStatus while registering. From RM's view, it should keep the latest updated container resource. So for above corner cases: 1. Result: container decreased 2. Result: container increased 3. Result: container decreased (because the latest resource AM sent to RM is decrese). So in RM side, it will check: if (rm.version >= nm.version) { // keep existing container in RM unchanged, and tell NM about this // why include "==" here is, if rm.version == nm.version, corner case #3 happened. } else { // change container in RM } So in summary what we need in protocol is: Container-version in ContainerTokenIdentifier COntainer-version in NMContainerStatus add a IncreasedContainer of NM-RM heartbeat, and include container-version in IncreasedContainer. Thoughts? MENG DING
        Hide
        jianhe Jian He added a comment -

        After thinking more, we may not need a version Id. Below is my suggestion:
        1) still keep the separate increasedContainers in the protocol
        2) If increased-container > RMContainer, RM ignores this.
        3) For the race condition on RM recovery, we may fix this by synchronizing ContainerMangagerImpl#increaseContainer and NM register call.

        Show
        jianhe Jian He added a comment - After thinking more, we may not need a version Id. Below is my suggestion: 1) still keep the separate increasedContainers in the protocol 2) If increased-container > RMContainer, RM ignores this. 3) For the race condition on RM recovery, we may fix this by synchronizing ContainerMangagerImpl#increaseContainer and NM register call.
        Hide
        mding MENG DING added a comment -

        Thanks a lot Wangda Tan and Jian He for your comments and suggestions.

        After more thoughts, I prefer Jian He's suggestion to synchronize ContainerMangagerImpl#increaseContainersResource with NM-RM registration. If we do that, we should be able to resolve the RM recovery race condition issue, more specifically:

        • If increaseContainersResource happens first, then container resource will be increased in NM before NM-RM registration.
        • If NM-RM registration happens first, then NM will get a new RM identifier after registration. Any subsequent increase request with a token issued by old RM will be rejected.

        For implementation, I think I can simply synchronize on the NMContext object in both ContainerMangagerImpl and NodeStatusUpdaterImpl.

        Let me know if you have further thoughts or comments. I am also wondering if we should do the same for ContainerMangagerImpl#startContainers?

        Show
        mding MENG DING added a comment - Thanks a lot Wangda Tan and Jian He for your comments and suggestions. After more thoughts, I prefer Jian He 's suggestion to synchronize ContainerMangagerImpl#increaseContainersResource with NM-RM registration. If we do that, we should be able to resolve the RM recovery race condition issue, more specifically: If increaseContainersResource happens first, then container resource will be increased in NM before NM-RM registration. If NM-RM registration happens first, then NM will get a new RM identifier after registration. Any subsequent increase request with a token issued by old RM will be rejected. For implementation, I think I can simply synchronize on the NMContext object in both ContainerMangagerImpl and NodeStatusUpdaterImpl . Let me know if you have further thoughts or comments. I am also wondering if we should do the same for ContainerMangagerImpl#startContainers ?
        Hide
        jianhe Jian He added a comment -

        I am also wondering if we should do the same for ContainerMangagerImpl#startContainers

        That should be the same issue. We may do this too.

        Show
        jianhe Jian He added a comment - I am also wondering if we should do the same for ContainerMangagerImpl#startContainers That should be the same issue. We may do this too.
        Hide
        mding MENG DING added a comment -

        Attaching latest patch that addresses the following:

        • Keep the increasedContainers in the protocol
        • Synchronize ContainerMangagerImpl#startContainers and ContainerMangagerImpl#increaseContainersResource with NodeStatusUpdaterImpl#registerWithRM. The lock scope in registerWithRM is minimized to cover only the resourceTracker.registerNodeManager(request) call
        • Add a test case to test synchronization between increaseContainersResource and registerWithRM
        Show
        mding MENG DING added a comment - Attaching latest patch that addresses the following: Keep the increasedContainers in the protocol Synchronize ContainerMangagerImpl#startContainers and ContainerMangagerImpl#increaseContainersResource with NodeStatusUpdaterImpl#registerWithRM . The lock scope in registerWithRM is minimized to cover only the resourceTracker.registerNodeManager(request) call Add a test case to test synchronization between increaseContainersResource and registerWithRM
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 17s Findbugs (version ) appears to be broken on YARN-1197.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 8m 7s There were no new javac warning messages.
        +1 javadoc 10m 29s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 7s There were no new checkstyle issues.
        +1 whitespace 0m 22s The patch has no lines that end in whitespace.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        -1 findbugs 3m 19s The patch appears to introduce 7 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 6m 58s Tests passed in hadoop-yarn-client.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
        +1 yarn tests 6m 34s Tests passed in hadoop-yarn-server-nodemanager.
            57m 11s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-common



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12751301/YARN-1644-YARN-1197.6.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision YARN-1197 / 1d1905e
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
        hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/testrun_hadoop-yarn-client.txt
        hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8884/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8884/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 17s Findbugs (version ) appears to be broken on YARN-1197 . +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 7s There were no new javac warning messages. +1 javadoc 10m 29s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 7s There were no new checkstyle issues. +1 whitespace 0m 22s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 3m 19s The patch appears to introduce 7 new Findbugs (version 3.0.0) warnings. +1 yarn tests 6m 58s Tests passed in hadoop-yarn-client. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 34s Tests passed in hadoop-yarn-server-nodemanager.     57m 11s   Reason Tests FindBugs module:hadoop-yarn-server-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751301/YARN-1644-YARN-1197.6.patch Optional Tests javac unit findbugs checkstyle javadoc git revision YARN-1197 / 1d1905e Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8884/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8884/console This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        MENG DING, thanks for updating. +1 latest patch

        is the findbugs warning related ?

        Show
        jianhe Jian He added a comment - MENG DING , thanks for updating. +1 latest patch is the findbugs warning related ?
        Hide
        mding MENG DING added a comment -

        The findbugs warning is not related. The link given shows 0 warnings:
        https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html.
        My own tests locally did not show any findbugs warnings.

        Thanks for the review!

        Show
        mding MENG DING added a comment - The findbugs warning is not related. The link given shows 0 warnings: https://builds.apache.org/job/PreCommit-YARN-Build/8884/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html . My own tests locally did not show any findbugs warnings. Thanks for the review!
        Hide
        jianhe Jian He added a comment -

        committed to YARN-1197 branch. thanks MENG DING !
        thanks Wangda Tan for reviewing !

        Show
        jianhe Jian He added a comment - committed to YARN-1197 branch. thanks MENG DING ! thanks Wangda Tan for reviewing !
        Hide
        leftnoteasy Wangda Tan added a comment -

        MENG DING, could you take a look at findbugs? I can reproduce it locally.

        You can run "mvn clean findbugs:findbugs" under yarn-server-common. Please open a ticket to track the findbugs fix if you can reproduce it.

        Show
        leftnoteasy Wangda Tan added a comment - MENG DING , could you take a look at findbugs? I can reproduce it locally. You can run "mvn clean findbugs:findbugs" under yarn-server-common. Please open a ticket to track the findbugs fix if you can reproduce it.
        Hide
        mding MENG DING added a comment -

        Hi, Wangda Tan

        There are 7 findbugs warnings, but they already existed before the patch. This patch does not generate new findbugs warnings.

        I took a quick look at some of the warnings:

        • The warnings in NodeStatusPBImpl are most likely because getContainersUtilization/setContainersUtilization/getNodeUtilization/setNodeUtilization are not synchronized
        • The warnings in WebServices are probably because of potential NPE.

        I will open a ticket to fix them.
        Meng

        Show
        mding MENG DING added a comment - Hi, Wangda Tan There are 7 findbugs warnings, but they already existed before the patch. This patch does not generate new findbugs warnings. I took a quick look at some of the warnings: The warnings in NodeStatusPBImpl are most likely because getContainersUtilization/setContainersUtilization/getNodeUtilization/setNodeUtilization are not synchronized The warnings in WebServices are probably because of potential NPE. I will open a ticket to fix them. Meng
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8505/)
        YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8505/ ) YARN-1644 . RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/)
        YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/ ) YARN-1644 . RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #429 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/429/)
        YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #429 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/429/ ) YARN-1644 . RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1169 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1169/)
        YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1169 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1169/ ) YARN-1644 . RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #409 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/409/)
        YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #409 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/409/ ) YARN-1644 . RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2375 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2375/)
        YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2375 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2375/ ) YARN-1644 . RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2348 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2348/)
        YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2348 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2348/ ) YARN-1644 . RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding (wangda: rev c3dc1af072574f5890a8d43e4d60526951b4b8bc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/CHANGES.txt

          People

          • Assignee:
            mding MENG DING
            Reporter:
            leftnoteasy Wangda Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development