Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1197 Support changing resources of an allocated container
  3. YARN-1643

Make ContainersMonitor can support change monitoring size of an allocated container in NM side

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. YARN-1643.1.patch
      20 kB
      Wangda Tan
    2. YARN-1643.2.patch
      23 kB
      MENG DING
    3. YARN-1643.3.patch
      23 kB
      MENG DING
    4. YARN-1643-YARN-1197.4.patch
      23 kB
      MENG DING
    5. YARN-1643-YARN-1197.5.patch
      51 kB
      MENG DING
    6. YARN-1643-YARN-1197.6.patch
      51 kB
      MENG DING
    7. YARN-1643-YARN-1197.7.patch
      37 kB
      MENG DING

      Issue Links

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        Added patch#1 to review

        Show
        leftnoteasy Wangda Tan added a comment - Added patch#1 to review
        Hide
        mding MENG DING added a comment -

        Attach patch for ContainersMonitorImpl for review.

        Currently the ContainersMonitorImpl starts up one monitoring thread, and by default checks started and stopped containers every 3 seconds.

        For resized containers, we cannot use the same logic as the delay is unacceptable.

        A wait/notify mechanism is implemented such that as soon as a CHANGE_MONITORING_CONTAINER_RESOURCE event is received, the monitor cycle is immediately triggered.

        Show
        mding MENG DING added a comment - Attach patch for ContainersMonitorImpl for review. Currently the ContainersMonitorImpl starts up one monitoring thread, and by default checks started and stopped containers every 3 seconds. For resized containers, we cannot use the same logic as the delay is unacceptable. A wait/notify mechanism is implemented such that as soon as a CHANGE_MONITORING_CONTAINER_RESOURCE event is received, the monitor cycle is immediately triggered.
        Hide
        mding MENG DING added a comment -

        Updated the patch to resolve the conflicts with YARN-1012 based on the latest rebase.

        Show
        mding MENG DING added a comment - Updated the patch to resolve the conflicts with YARN-1012 based on the latest rebase.
        Hide
        mding MENG DING added a comment -

        We have decided to update resource synchronously in YARN-3867 for now.

        First upload a patch (YARN-1643.4.patch) that does container monitor and resource enforcement asynchronously for reference, in case we need to support asynchronous resource update in the future.

        Show
        mding MENG DING added a comment - We have decided to update resource synchronously in YARN-3867 for now. First upload a patch ( YARN-1643 .4.patch) that does container monitor and resource enforcement asynchronously for reference, in case we need to support asynchronous resource update in the future.
        Hide
        mding MENG DING added a comment -

        Upload the patch that supports synchronous resource update in ContainersMonitorImpl.

        There are a number of changes:

        • Extract out the code from the MonitoringThread.run and put it into a common function enforceResourceLimits. This function will be called by both ContainersMonitorImpl's monitoring thread, and the ContainerManagerImpl thread, so it needs to be synchronized (in order to guard access to the trackingContainers object).
        • Call container.setResource directly to update container resource.
        • Add TestContainerManager#testChangeContainerResource and TestContainersMonitorResourceChange test cases.
        Show
        mding MENG DING added a comment - Upload the patch that supports synchronous resource update in ContainersMonitorImpl . There are a number of changes: Extract out the code from the MonitoringThread.run and put it into a common function enforceResourceLimits . This function will be called by both ContainersMonitorImpl's monitoring thread, and the ContainerManagerImpl thread, so it needs to be synchronized (in order to guard access to the trackingContainers object). Call container.setResource directly to update container resource. Add TestContainerManager#testChangeContainerResource and TestContainersMonitorResourceChange test cases.
        Hide
        mding MENG DING added a comment -

        Update the patch to fix build issue as YARN-1197 branch has been rebased.

        Show
        mding MENG DING added a comment - Update the patch to fix build issue as YARN-1197 branch has been rebased.
        Hide
        jianhe Jian He added a comment -

        Thanks MENG DING !

        • Here, memory * 2^20, but it gets reverted later on at info.pmemLimit >> 20, we can just use the original value ?
                this.pmemLimit = resource.getMemory() * 1024L * 1024L;
          
        • Do you think we can change the trackingContainers to be concurrentHashMap and update the ptInfo directly ? Also the getter and setter of ptInfo can synchronize on the ptInfo object
                synchronized (this.containersToBeChanged) {
                  this.containersToBeChanged.add(new ContainerResourceToChange(
                      containerId, changeEvent.getResource()));
                }
                // Trigger a round of containers check immediately
                enforceResourceLimits();
          
        Show
        jianhe Jian He added a comment - Thanks MENG DING ! Here, memory * 2^20, but it gets reverted later on at info.pmemLimit >> 20 , we can just use the original value ? this .pmemLimit = resource.getMemory() * 1024L * 1024L; Do you think we can change the trackingContainers to be concurrentHashMap and update the ptInfo directly ? Also the getter and setter of ptInfo can synchronize on the ptInfo object synchronized ( this .containersToBeChanged) { this .containersToBeChanged.add( new ContainerResourceToChange( containerId, changeEvent.getResource())); } // Trigger a round of containers check immediately enforceResourceLimits();
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 15m 30s Findbugs (version ) appears to be broken on YARN-1197.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 7m 46s There were no new javac warning messages.
        +1 javadoc 9m 42s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 20s There were no new checkstyle issues.
        +1 whitespace 0m 5s The patch has no lines that end in whitespace.
        +1 install 1m 23s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 1m 14s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 6m 20s Tests failed in hadoop-yarn-server-nodemanager.
            43m 21s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-nodemanager
        Failed unit tests hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12747666/YARN-1643-YARN-1197.6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision YARN-1197 / cb95662
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8700/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8700/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8700/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8700/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 30s Findbugs (version ) appears to be broken on YARN-1197 . +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 20s There were no new checkstyle issues. +1 whitespace 0m 5s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 1m 14s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. -1 yarn tests 6m 20s Tests failed in hadoop-yarn-server-nodemanager.     43m 21s   Reason Tests FindBugs module:hadoop-yarn-server-nodemanager Failed unit tests hadoop.yarn.server.nodemanager.TestContainerManagerWithLCE Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747666/YARN-1643-YARN-1197.6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision YARN-1197 / cb95662 Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8700/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8700/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8700/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8700/console This message was automatically generated.
        Hide
        mding MENG DING added a comment -

        Thanks Jian He for the review:

        Here, memory * 2^20, but it gets reverted later on at info.pmemLimit >> 20, we can just use the original value ?

        Will do.

        Do you think we can change the trackingContainers to be concurrentHashMap and update the ptInfo directly ? Also the getter and setter of ptInfo can synchronize on the ptInfo object

        Yes, we can make trackingContainers a ConcurrentHashMap, and add setter to ProcessTreeInfo for vemLimit, pmemLimit, and cpuVcores, then have the getter and setter synchronized on the object.

        IIUC, the main benefit is that we don't need to synchronize on the enforceResourceLimits call, which can be heavy, right? If that is the case, we probably also need to have proper synchronization for ResourceCalculatorProcessTree, e.g., ProcfsBasedProcessTree/WindowsBasedProcessTree? These objects could be updated by multiple threads as well. I was afraid that the code change may be too much?

        For other objects like containersToBeChanged, containersToBeAdded, containersToBeRemoved, I think we still need to synchronize on the entire map like the way it is right now, because we are calling functions like containersToBeRemoved.clear().

        Thoughts?

        Show
        mding MENG DING added a comment - Thanks Jian He for the review: Here, memory * 2^20, but it gets reverted later on at info.pmemLimit >> 20, we can just use the original value ? Will do. Do you think we can change the trackingContainers to be concurrentHashMap and update the ptInfo directly ? Also the getter and setter of ptInfo can synchronize on the ptInfo object Yes, we can make trackingContainers a ConcurrentHashMap , and add setter to ProcessTreeInfo for vemLimit, pmemLimit, and cpuVcores, then have the getter and setter synchronized on the object. IIUC, the main benefit is that we don't need to synchronize on the enforceResourceLimits call, which can be heavy, right? If that is the case, we probably also need to have proper synchronization for ResourceCalculatorProcessTree , e.g., ProcfsBasedProcessTree / WindowsBasedProcessTree ? These objects could be updated by multiple threads as well. I was afraid that the code change may be too much? For other objects like containersToBeChanged , containersToBeAdded , containersToBeRemoved , I think we still need to synchronize on the entire map like the way it is right now, because we are calling functions like containersToBeRemoved.clear() . Thoughts?
        Hide
        mding MENG DING added a comment -

        IIUC, the main benefit is that we don't need to synchronize on the enforceResourceLimits call, which can be heavy, right? If that is the case, we probably also need to have proper synchronization for ResourceCalculatorProcessTree, e.g., ProcfsBasedProcessTree/WindowsBasedProcessTree? These objects could be updated by multiple threads as well. I was afraid that the code change may be too much?

        I think I find a way without having to synchronize on ResourceCalculatorProcessTree. All that is needed for synchronization in this class is the trackingContainers map and the access to the vmemLimit/pmemLimit/cpuVcores fields. The actual resource limit enforcement can still be handled in the MonitoringThread.run thread only.

        Show
        mding MENG DING added a comment - IIUC, the main benefit is that we don't need to synchronize on the enforceResourceLimits call, which can be heavy, right? If that is the case, we probably also need to have proper synchronization for ResourceCalculatorProcessTree, e.g., ProcfsBasedProcessTree/WindowsBasedProcessTree? These objects could be updated by multiple threads as well. I was afraid that the code change may be too much? I think I find a way without having to synchronize on ResourceCalculatorProcessTree . All that is needed for synchronization in this class is the trackingContainers map and the access to the vmemLimit/pmemLimit/cpuVcores fields. The actual resource limit enforcement can still be handled in the MonitoringThread.run thread only.
        Hide
        mding MENG DING added a comment -

        Attaching latest patch which addressed the following:

        • Make trackingContainers a ConcurrentHashMap, and remove containersToBeRemoved and containersToBeAdded and corresponding logic. Containers are directly added to/removed from/updated in trackingContainers when corresponding events are received.
        • Synchronize getters and setters in ProcessTreeInfo with regard to the vmemLimit/pmemLimit/cpuVcores fields.
        • Previous patch didn't handle container metrics update for container resize. Add it and extract container metrics logic into a common function.
        • Add relevant test cases
        Show
        mding MENG DING added a comment - Attaching latest patch which addressed the following: Make trackingContainers a ConcurrentHashMap, and remove containersToBeRemoved and containersToBeAdded and corresponding logic. Containers are directly added to/removed from/updated in trackingContainers when corresponding events are received. Synchronize getters and setters in ProcessTreeInfo with regard to the vmemLimit/pmemLimit/cpuVcores fields. Previous patch didn't handle container metrics update for container resize. Add it and extract container metrics logic into a common function. Add relevant test cases
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 15m 14s Findbugs (version ) appears to be broken on YARN-1197.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
        +1 javac 7m 42s There were no new javac warning messages.
        +1 javadoc 9m 41s There were no new javadoc warning messages.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 20s There were no new checkstyle issues.
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 20s mvn install still works.
        +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
        +1 findbugs 1m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 6m 14s Tests failed in hadoop-yarn-server-nodemanager.
            42m 42s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12748018/YARN-1643-YARN-1197.7.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision YARN-1197 / cb95662
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8716/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8716/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8716/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 14s Findbugs (version ) appears to be broken on YARN-1197 . +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 20s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 1m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 6m 14s Tests failed in hadoop-yarn-server-nodemanager.     42m 42s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748018/YARN-1643-YARN-1197.7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision YARN-1197 / cb95662 hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8716/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8716/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8716/console This message was automatically generated.
        Hide
        mding MENG DING added a comment -

        The failed test does not seem to be related.

        Show
        mding MENG DING added a comment - The failed test does not seem to be related.
        Hide
        jianhe Jian He added a comment -

        Committed to YARN-1197 branch, thanks Meng and Wangda !

        Show
        jianhe Jian He added a comment - Committed to YARN-1197 branch, thanks Meng and Wangda !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8505/)
        YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8505/ ) YARN-1643 . Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/)
        YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/ ) YARN-1643 . Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #429 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/429/)
        YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #429 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/429/ ) YARN-1643 . Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1169 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1169/)
        YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1169 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1169/ ) YARN-1643 . Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #409 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/409/)
        YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #409 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/409/ ) YARN-1643 . Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2375 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2375/)
        YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2375 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2375/ ) YARN-1643 . Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2348 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2348/)
        YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2348 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2348/ ) YARN-1643 . Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan (wangda: rev c59ae4eeb17e52e8fc659b9962d20628719fc621) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorPlugin.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/MockResourceCalculatorProcessTree.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java

          People

          • Assignee:
            mding MENG DING
            Reporter:
            leftnoteasy Wangda Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development