Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.9.0
    • 3.2.0
    • nodemanager
    • Reviewed

    Description

      YARN-1856 adds memory cgroups enforcing support. We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage. Today, one could set the swappiness to control this, but clusters with swap turned off exist.

      Attachments

        1. YARN-4599.016.patch
          117 kB
          Miklos Szegedi
        2. YARN-4599.015.patch
          117 kB
          Miklos Szegedi
        3. YARN-4599.014.patch
          117 kB
          Miklos Szegedi
        4. YARN-4599.013.patch
          116 kB
          Miklos Szegedi
        5. YARN-4599.012.patch
          116 kB
          Miklos Szegedi
        6. YARN-4599.011.patch
          116 kB
          Miklos Szegedi
        7. YARN-4599.010.patch
          116 kB
          Miklos Szegedi
        8. YARN-4599.009.patch
          116 kB
          Miklos Szegedi
        9. YARN-4599.008.patch
          115 kB
          Miklos Szegedi
        10. YARN-4599.007.patch
          114 kB
          Miklos Szegedi
        11. Elastic Memory Control in YARN.pdf
          92 kB
          Miklos Szegedi
        12. YARN-4599.006.patch
          102 kB
          Miklos Szegedi
        13. YARN-4599.005.patch
          101 kB
          Miklos Szegedi
        14. YARN-4599.004.patch
          101 kB
          Miklos Szegedi
        15. YARN-4599.003.patch
          98 kB
          Miklos Szegedi
        16. YARN-4599.002.patch
          96 kB
          Miklos Szegedi
        17. YARN-4599.001.patch
          96 kB
          Miklos Szegedi
        18. YARN-4599.000.patch
          95 kB
          Miklos Szegedi
        19. YARN-4599.sandflee.patch
          33 kB
          sandflee
        20. yarn-4599-not-so-useful.patch
          9 kB
          Karthik Kambatla

        Issue Links

          Activity

            We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage.

            We need this as something that can be set and I'd propose that the default be off given that administrators are going to be very confused when they see container usage go above the limit.

            FWIW, we have jobs running on our clusters that the NM's memory control doesn't stop from breaking nodes because it is too slow.

            aw Allen Wittenauer added a comment - We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage. We need this as something that can be set and I'd propose that the default be off given that administrators are going to be very confused when they see container usage go above the limit. FWIW, we have jobs running on our clusters that the NM's memory control doesn't stop from breaking nodes because it is too slow.
            mding MENG DING added a comment -

            I have a question regarding when OOM controller will kick in. Is it true that:

            • If memory.swappiness is set to 0, then OOM controller will kick in when memory.limit_in_bytes (hard limit) is reached.
            • If memory.swappiness is not set to 0, then OOM controller will kick in only when all available swap in the system is used up, as memory.memsw.limit_in_bytes is not a configurable parameter in YARN right now?

            Looking at this link, I am a little bit confused: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html

            Consider the following example: setting memory.limit_in_bytes = 2G and memory.memsw.limit_in_bytes = 4G for a certain cgroup will allow processes in that cgroup to allocate 2 GB of memory and, once exhausted, allocate another 2 GB of swap only. The memory.memsw.limit_in_bytes parameter represents the sum of memory and swap. Processes in a cgroup that does not have the memory.memsw.limit_in_bytes parameter set can potentially use up all the available swap (after exhausting the set memory limitation) and trigger an Out Of Memory situation caused by the lack of available swap.

            memory.swappiness Note that a value of 0 does not prevent process memory being swapped out; swap out might still happen when there is a shortage of system memory because the global virtual memory management logic does not read the cgroup value.

            mding MENG DING added a comment - I have a question regarding when OOM controller will kick in. Is it true that: If memory.swappiness is set to 0, then OOM controller will kick in when memory.limit_in_bytes (hard limit) is reached. If memory.swappiness is not set to 0, then OOM controller will kick in only when all available swap in the system is used up, as memory.memsw.limit_in_bytes is not a configurable parameter in YARN right now? Looking at this link, I am a little bit confused: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html Consider the following example: setting memory.limit_in_bytes = 2G and memory.memsw.limit_in_bytes = 4G for a certain cgroup will allow processes in that cgroup to allocate 2 GB of memory and, once exhausted, allocate another 2 GB of swap only. The memory.memsw.limit_in_bytes parameter represents the sum of memory and swap. Processes in a cgroup that does not have the memory.memsw.limit_in_bytes parameter set can potentially use up all the available swap (after exhausting the set memory limitation) and trigger an Out Of Memory situation caused by the lack of available swap. memory.swappiness Note that a value of 0 does not prevent process memory being swapped out; swap out might still happen when there is a shortage of system memory because the global virtual memory management logic does not read the cgroup value.
            mding MENG DING added a comment -

            aw:

            We need this as something that can be set and I'd propose that the default be off given that administrators are going to be very confused when they see container usage go above the limit.

            I thought the container process will be paused by the OOM controller when the limit is reached, so why will the memory usage go above the limit?

            mding MENG DING added a comment - aw : We need this as something that can be set and I'd propose that the default be off given that administrators are going to be very confused when they see container usage go above the limit. I thought the container process will be paused by the OOM controller when the limit is reached, so why will the memory usage go above the limit?

            Looked more into this, specifically on how to resume paused tasks. Per this article

            This operation is only allowed to the top cgroup of a sub-hierarchy.
            If OOM-killer is disabled, tasks under cgroup will hang/sleep
            in memory cgroup's OOM-waitqueue when they request accountable memory.
            
            For running them, you have to relax the memory cgroup's OOM status by
            	* enlarge limit or reduce usage.
            To reduce usage,
            	* kill some tasks.
            	* move some tasks to other group with account migration.
            	* remove some files (on tmpfs?)
            
            Then, stopped tasks will work again.
            
            At reading, current status of OOM is shown.
            	oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)
            	under_oom	 0 or 1 (if 1, the memory cgroup is under OOM, tasks may
            				 be stopped.)
            

            Looks like setting OOM control per each task is not particularly useful. We are likely better off setting hard limit for all yarn containers so they don't interfere anything else on the machine. We could disable OOM control on the cgroup corresponding to all yarn containers (not including NM) and if all containers are paused, the NM can decide what tasks to kill. This is particularly useful if we are oversubscribing the node.

            aw, vvasudev, vinodkv - what do you think?

            kasha Karthik Kambatla added a comment - Looked more into this, specifically on how to resume paused tasks. Per this article This operation is only allowed to the top cgroup of a sub-hierarchy. If OOM-killer is disabled, tasks under cgroup will hang/sleep in memory cgroup's OOM-waitqueue when they request accountable memory. For running them, you have to relax the memory cgroup's OOM status by * enlarge limit or reduce usage. To reduce usage, * kill some tasks. * move some tasks to other group with account migration. * remove some files (on tmpfs?) Then, stopped tasks will work again. At reading, current status of OOM is shown. oom_kill_disable 0 or 1 (if 1, oom-killer is disabled) under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may be stopped.) Looks like setting OOM control per each task is not particularly useful. We are likely better off setting hard limit for all yarn containers so they don't interfere anything else on the machine. We could disable OOM control on the cgroup corresponding to all yarn containers (not including NM) and if all containers are paused, the NM can decide what tasks to kill. This is particularly useful if we are oversubscribing the node. aw , vvasudev , vinodkv - what do you think?

            FWIW, just posted the not useful version of the patch.

            kasha Karthik Kambatla added a comment - FWIW, just posted the not useful version of the patch.

            We are likely better off setting hard limit for all yarn containers so they don't interfere anything else on the machine. We could disable OOM control on the cgroup corresponding to all yarn containers (not including NM) and if all containers are paused, the NM can decide what tasks to kill. This is particularly useful if we are oversubscribing the node.

            This seems like our only choice, given that none of the options to recover (when the per-container-limit is hit and when OOM-killer is disabled) are usable in practice for YARN containers.

            /cc sidharta-s, shanekumpf

            vinodkv Vinod Kumar Vavilapalli added a comment - We are likely better off setting hard limit for all yarn containers so they don't interfere anything else on the machine. We could disable OOM control on the cgroup corresponding to all yarn containers (not including NM) and if all containers are paused, the NM can decide what tasks to kill. This is particularly useful if we are oversubscribing the node. This seems like our only choice, given that none of the options to recover (when the per-container-limit is hit and when OOM-killer is disabled) are usable in practice for YARN containers. /cc sidharta-s , shanekumpf

            We are likely better off setting hard limit for all yarn containers so they don't interfere anything else on the machine. We could disable OOM control on the cgroup corresponding to all yarn containers (not including NM) and if all containers are paused, the NM can decide what tasks to kill. This is particularly useful if we are oversubscribing the node.

            kasha By tasks, do you mean entire containers? How would the NM know which containers are paused ? We would probably need to hook into the OOM notification mechanism for this. I am not sure how well that works in practice. That being said, it would be good to know that the container got killed because of an OOM event - otherwise it would be pretty hard to debug applications that run into this. (I think we might have hit this already with some internal testing with the memory cgroup)

            sidharta-s Sidharta Seethana added a comment - We are likely better off setting hard limit for all yarn containers so they don't interfere anything else on the machine. We could disable OOM control on the cgroup corresponding to all yarn containers (not including NM) and if all containers are paused, the NM can decide what tasks to kill. This is particularly useful if we are oversubscribing the node. kasha By tasks, do you mean entire containers? How would the NM know which containers are paused ? We would probably need to hook into the OOM notification mechanism for this. I am not sure how well that works in practice. That being said, it would be good to know that the container got killed because of an OOM event - otherwise it would be pretty hard to debug applications that run into this. (I think we might have hit this already with some internal testing with the memory cgroup)

            Yes, I meant containers.

            My proposal was to have a single parent cgroup for all Yarn containers. When that goes over the hard limit, all containers are paused. The NM should identify a victim for preemption, it could use different heuristics to prioritize between different containers all over their individual limits.

            kasha Karthik Kambatla added a comment - Yes, I meant containers. My proposal was to have a single parent cgroup for all Yarn containers. When that goes over the hard limit, all containers are paused. The NM should identify a victim for preemption, it could use different heuristics to prioritize between different containers all over their individual limits.
            sandflee sandflee added a comment -

            our cluster had implemented this, the solution is similar to kasha, we make a parent cgroup, disable oom_control , enable hierarchy, setting hard limit to nm total resource, so containers could continue to run util the resource of all container used exceed the hardlimit, nm will be notified and pick one container to kill.

            sandflee sandflee added a comment - our cluster had implemented this, the solution is similar to kasha , we make a parent cgroup, disable oom_control , enable hierarchy, setting hard limit to nm total resource, so containers could continue to run util the resource of all container used exceed the hardlimit, nm will be notified and pick one container to kill.

            sandflee - more details on your experience would help. Any metrics on how often the NM has to intervene? And, how long does it take for the NM to unblock containers once everything is paused?

            kasha Karthik Kambatla added a comment - sandflee - more details on your experience would help. Any metrics on how often the NM has to intervene? And, how long does it take for the NM to unblock containers once everything is paused?

            Oh, and any interest in contributing to do this natively in YARN?

            kasha Karthik Kambatla added a comment - Oh, and any interest in contributing to do this natively in YARN?
            sandflee sandflee added a comment -

            Thanks kasha,

            Any metrics on how often the NM has to intervene?

            it depends on the business of NM and how user set container memory. In our cluster with about 5000 NM, average of the NM resource allocation rate is 90%, the actually mem usage rate is 60%. most user request memory more than container really use. without this improvement, several thousands container oom kill event happens, with the fix, not more than ten oom event happens.

            how long does it take for the NM to unblock containers once everything is paused ?

            we payed not much attention on this, usually nm handle oom event very quickly, the most time maybe used by kernel to free pagecache/container mem, but we didn't stat this.

            Oh, and any interest in contributing to do this natively in YARN?

            yes, I'd like to update our patch in one or two weeks.

            sandflee sandflee added a comment - Thanks kasha , Any metrics on how often the NM has to intervene? it depends on the business of NM and how user set container memory. In our cluster with about 5000 NM, average of the NM resource allocation rate is 90%, the actually mem usage rate is 60%. most user request memory more than container really use. without this improvement, several thousands container oom kill event happens, with the fix, not more than ten oom event happens. how long does it take for the NM to unblock containers once everything is paused ? we payed not much attention on this, usually nm handle oom event very quickly, the most time maybe used by kernel to free pagecache/container mem, but we didn't stat this. Oh, and any interest in contributing to do this natively in YARN? yes, I'd like to update our patch in one or two weeks.

            yes, I'd like to update our patch in one or two weeks.

            Great to hear. Happy to help with reviews and discussion as required.

            kasha Karthik Kambatla added a comment - yes, I'd like to update our patch in one or two weeks. Great to hear. Happy to help with reviews and discussion as required.
            sandflee sandflee added a comment -

            Hi kasha, sorry for the delay, update a initial patch to review.

            sandflee sandflee added a comment - Hi kasha , sorry for the delay, update a initial patch to review.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 17s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
            0 mvndep 0m 11s Maven dependency ordering for branch
            +1 mvninstall 6m 0s trunk passed
            +1 compile 6m 15s trunk passed
            +1 checkstyle 1m 24s trunk passed
            +1 mvnsite 1m 30s trunk passed
            +1 mvneclipse 0m 44s trunk passed
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
            +1 findbugs 2m 26s trunk passed
            +1 javadoc 1m 9s trunk passed
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 14s the patch passed
            +1 compile 6m 13s the patch passed
            +1 cc 6m 13s the patch passed
            -1 javac 6m 13s root generated 2 new + 691 unchanged - 0 fixed = 693 total (was 691)
            -1 checkstyle 1m 24s root: The patch generated 19 new + 289 unchanged - 0 fixed = 308 total (was 289)
            +1 mvnsite 1m 31s the patch passed
            +1 mvneclipse 0m 44s the patch passed
            -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
            +1 xml 0m 3s The patch has no ill-formed XML file.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
            -1 findbugs 0m 48s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
            -1 javadoc 0m 17s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 3 new + 279 unchanged - 0 fixed = 282 total (was 279)
            +1 unit 0m 9s hadoop-assemblies in the patch passed.
            +1 unit 0m 23s hadoop-yarn-api in the patch passed.
            +1 unit 2m 8s hadoop-yarn-common in the patch passed.
            +1 unit 13m 1s hadoop-yarn-server-nodemanager in the patch passed.
            -1 asflicense 0m 21s The patch generated 1 ASF License warnings.
            52m 17s



            Reason Tests
            FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
              Boxing/unboxing to parse a primitive org.apache.hadoop.yarn.server.nodemanager.containermanager.CgroupsOOMHandler$MonitoringThread.selectContainer(Map) At CgroupsOOMHandler.java:org.apache.hadoop.yarn.server.nodemanager.containermanager.CgroupsOOMHandler$MonitoringThread.selectContainer(Map) At CgroupsOOMHandler.java:[line 208]



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:2c91fd8
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810194/YARN-4599.sandflee.patch
            JIRA Issue YARN-4599
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle cc
            uname Linux 031f3d40559e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 8e8cb4c
            Default Java 1.8.0_91
            findbugs v3.0.0
            javac https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/diff-compile-javac-root.txt
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/diff-checkstyle-root.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/whitespace-eol.txt
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12009/testReport/
            asflicense https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/patch-asflicense-problems.txt
            modules C: hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/12009/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 0s trunk passed +1 compile 6m 15s trunk passed +1 checkstyle 1m 24s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 44s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 2m 26s trunk passed +1 javadoc 1m 9s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 6m 13s the patch passed +1 cc 6m 13s the patch passed -1 javac 6m 13s root generated 2 new + 691 unchanged - 0 fixed = 693 total (was 691) -1 checkstyle 1m 24s root: The patch generated 19 new + 289 unchanged - 0 fixed = 308 total (was 289) +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 44s the patch passed -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies -1 findbugs 0m 48s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 17s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 3 new + 279 unchanged - 0 fixed = 282 total (was 279) +1 unit 0m 9s hadoop-assemblies in the patch passed. +1 unit 0m 23s hadoop-yarn-api in the patch passed. +1 unit 2m 8s hadoop-yarn-common in the patch passed. +1 unit 13m 1s hadoop-yarn-server-nodemanager in the patch passed. -1 asflicense 0m 21s The patch generated 1 ASF License warnings. 52m 17s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   Boxing/unboxing to parse a primitive org.apache.hadoop.yarn.server.nodemanager.containermanager.CgroupsOOMHandler$MonitoringThread.selectContainer(Map) At CgroupsOOMHandler.java:org.apache.hadoop.yarn.server.nodemanager.containermanager.CgroupsOOMHandler$MonitoringThread.selectContainer(Map) At CgroupsOOMHandler.java: [line 208] Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12810194/YARN-4599.sandflee.patch JIRA Issue YARN-4599 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle cc uname Linux 031f3d40559e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8e8cb4c Default Java 1.8.0_91 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12009/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/12009/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-assemblies hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/12009/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            Thank you for the patch sandflee. This would be useful for many users.
            I have a few questions.
            1. CGroupsElasticMemoryResourceHandlerImpl copies lots of code from CGroupsMemoryResourceHandlerImpl. Have you considered inheritance to reduce redundancy?
            2. Is it better to use JNI as in the current patch, or implement the event logic in Linux Container Executor?
            3. Could you please rebase it? It does not build anymore.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you for the patch sandflee . This would be useful for many users. I have a few questions. 1. CGroupsElasticMemoryResourceHandlerImpl copies lots of code from CGroupsMemoryResourceHandlerImpl. Have you considered inheritance to reduce redundancy? 2. Is it better to use JNI as in the current patch, or implement the event logic in Linux Container Executor? 3. Could you please rebase it? It does not build anymore.
            sandflee sandflee added a comment -

            Hi, miklos.szegedi@cloudera.com, I think the first thing is whether community accept this propose, if yes, we could push forward, if not it will be a burden to keep the patch with the trunk. Thought?
            use Linux Container Executor seems simple the code but will use a new process additionally, I'm ok to both.

            sandflee sandflee added a comment - Hi, miklos.szegedi@cloudera.com , I think the first thing is whether community accept this propose, if yes, we could push forward, if not it will be a burden to keep the patch with the trunk. Thought? use Linux Container Executor seems simple the code but will use a new process additionally, I'm ok to both.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            Thank you, sandflee for the reply. The patch looks good to me in general, if others also like it.
            Indeed LCE is a new process, maybe we could just poll the memory.oom_control file? I am not against JNI but this is the first time it is used in the node manager.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you, sandflee for the reply. The patch looks good to me in general, if others also like it. Indeed LCE is a new process, maybe we could just poll the memory.oom_control file? I am not against JNI but this is the first time it is used in the node manager.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            sandflee, do you plan to continue working on the patch? I have a similar one, YARN-6677 and it would be great to coordinate. I think it would be better to do the event logic in a separate application detached from the java process.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - sandflee , do you plan to continue working on the patch? I have a similar one, YARN-6677 and it would be great to coordinate. I think it would be better to do the event logic in a separate application detached from the java process.
            sandflee sandflee added a comment -

            Hi, miklos.szegedi@cloudera.com, I'm busy on other work recently , feel free to assign this to yourself, will join you when not so busy.

            sandflee sandflee added a comment - Hi, miklos.szegedi@cloudera.com , I'm busy on other work recently , feel free to assign this to yourself, will join you when not so busy.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            Thank you, sandflee. I am working on a patch now.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you, sandflee . I am working on a patch now.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            I will provide the patch shortly. Here are the design suggestions of the patch:

            • The basic idea is what was discussed above. It disables the OOM killer on the hadoop-yarn cgroup. This will trigger a pause on all containers when all of them exceeded the node limit.
            • YARN will be notified with an executable listening to the cgroups OOM Linux event. This should be very fast. The executable is oom-listener, not container-executor. This is because it does not need to run as root. I avoided JNI to be more defensive on security and it also helps to test the executable easier.
            • When YARN receives the notification, it runs a pluggable OOM handler to resolve the situation. YARN is outside the hadoop-yarn group, so it can run freely, all containers are frozen at this point. Different users may have different preferences, thus the handler is pluggable.
            • The default OOM handler picks the latest container that ran out of it's request. This ensures that it kills a container that did not cost much so far but it keeps guaranteed containers that play by the rules and use memory within their limits. It repeats the process until the OOM is resolved. Based on my experiments the kernel updates the flag almost instantaneously, so it will just kill as many containers as necessary.
            • If the default OOM handler cannot pick a container with the logic above it kills the latest container until the OOM is resolved.
            • If we are still in OOM without any containers, an exception is thrown and the node is brought down. This can be the case, if containers leaked processes, had processes running with another user and cannot be killed the container user or someone put a process into the root hadoop-yarn cgroup.
            • The killer is not the normal container cleanup code. The standard behaviour is to send a SIGTERM to the container PGID process and if it does not respond in 250 milliseconds, it sends a SIGKILL. However, in our case all the processes are frozen by cgroups, so that they cannot respond to a SIGTERM. Because of this it uses the standard container executor code to send a SIGKILL to the PGID right away with the container user. The kernel OOM killer would do the same. This works pretty fast. It walks through all the thread/process IDs in the tasks file, so that all active PGIDs are found in the container. The current code does not delete standalone processes that are not a process group leader. If they are not part of one of the container local process groups they may be leaked. It also cannot handle processes that are running as different users in than the container user in the cgroup of the container. This should be rare.
            • The code adds a watchdog to measure the time to resolve an OOM situation. The time to resolve an OOM situation takes 10-160 milliseconds based on my experiments.
            • The patch contains documentation to set up and troubleshoot the feature.
            • I was able to test it manually but I did not do large scale and longhaul tests, yet.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - I will provide the patch shortly. Here are the design suggestions of the patch: The basic idea is what was discussed above. It disables the OOM killer on the hadoop-yarn cgroup. This will trigger a pause on all containers when all of them exceeded the node limit. YARN will be notified with an executable listening to the cgroups OOM Linux event. This should be very fast. The executable is oom-listener, not container-executor. This is because it does not need to run as root. I avoided JNI to be more defensive on security and it also helps to test the executable easier. When YARN receives the notification, it runs a pluggable OOM handler to resolve the situation. YARN is outside the hadoop-yarn group, so it can run freely, all containers are frozen at this point. Different users may have different preferences, thus the handler is pluggable. The default OOM handler picks the latest container that ran out of it's request. This ensures that it kills a container that did not cost much so far but it keeps guaranteed containers that play by the rules and use memory within their limits. It repeats the process until the OOM is resolved. Based on my experiments the kernel updates the flag almost instantaneously, so it will just kill as many containers as necessary. If the default OOM handler cannot pick a container with the logic above it kills the latest container until the OOM is resolved. If we are still in OOM without any containers, an exception is thrown and the node is brought down. This can be the case, if containers leaked processes, had processes running with another user and cannot be killed the container user or someone put a process into the root hadoop-yarn cgroup. The killer is not the normal container cleanup code. The standard behaviour is to send a SIGTERM to the container PGID process and if it does not respond in 250 milliseconds, it sends a SIGKILL. However, in our case all the processes are frozen by cgroups, so that they cannot respond to a SIGTERM. Because of this it uses the standard container executor code to send a SIGKILL to the PGID right away with the container user. The kernel OOM killer would do the same. This works pretty fast. It walks through all the thread/process IDs in the tasks file, so that all active PGIDs are found in the container. The current code does not delete standalone processes that are not a process group leader. If they are not part of one of the container local process groups they may be leaked. It also cannot handle processes that are running as different users in than the container user in the cgroup of the container. This should be rare. The code adds a watchdog to measure the time to resolve an OOM situation. The time to resolve an OOM situation takes 10-160 milliseconds based on my experiments. The patch contains documentation to set up and troubleshoot the feature. I was able to test it manually but I did not do large scale and longhaul tests, yet.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            aw, haibo.chenkasha, mding, sandflee, sidharta-s, vinodkv  do you have time to review the patch or do you have any comments on the design approach?

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - aw , haibo.chen ,  kasha , mding , sandflee , sidharta-s , vinodkv   do you have time to review the patch or do you have any comments on the design approach?
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 25s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  trunk Compile Tests
            0 mvndep 1m 33s Maven dependency ordering for branch
            +1 mvninstall 31m 37s trunk passed
            +1 compile 35m 25s trunk passed
            +1 checkstyle 3m 31s trunk passed
            +1 mvnsite 20m 19s trunk passed
            +1 shadedclient 34m 1s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 34s trunk passed
            +1 javadoc 5m 42s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            +1 mvninstall 27m 39s the patch passed
            +1 compile 28m 28s the patch passed
            -1 cc 28m 28s root generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
            +1 javac 28m 28s the patch passed
            -0 checkstyle 3m 35s root: The patch generated 17 new + 212 unchanged - 1 fixed = 229 total (was 213)
            +1 mvnsite 24m 0s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 11m 10s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            -1 findbugs 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
            +1 javadoc 6m 53s the patch passed
                  Other Tests
            -1 unit 16m 14s root in the patch failed.
            +1 asflicense 0m 41s The patch does not generate ASF License warnings.
            237m 48s



            Reason Tests
            FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
              Inconsistent synchronization of org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.stopped; locked 66% of time Unsynchronized access at CGroupElasticMemoryController.java:66% of time Unsynchronized access at CGroupElasticMemoryController.java:[line 242]
              Nullcheck of conf at line 360 of value previously dereferenced in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.getOOMListenerExecutablePath(Configuration) At CGroupElasticMemoryController.java:360 of value previously dereferenced in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.getOOMListenerExecutablePath(Configuration) At CGroupElasticMemoryController.java:[line 357]
              Exceptional return value of java.util.concurrent.ExecutorService.submit(Callable) ignored in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.run() At CGroupElasticMemoryController.java:ignored in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.run() At CGroupElasticMemoryController.java:[line 222]
              Self assignment of field CGroupElasticMemoryController.oomHandler in new org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController(Configuration, Context, CGroupsHandler, boolean, boolean, long) At CGroupElasticMemoryController.java:in new org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController(Configuration, Context, CGroupsHandler, boolean, boolean, long) At CGroupElasticMemoryController.java:[line 120]
            Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12921279/YARN-4599.000.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux a1f15c6c07fb 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / a3b416f
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            cc https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/diff-compile-cc-root.txt
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/diff-checkstyle-root.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/whitespace-eol.txt
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20587/testReport/
            Max. process+thread count 1496 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20587/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 1m 33s Maven dependency ordering for branch +1 mvninstall 31m 37s trunk passed +1 compile 35m 25s trunk passed +1 checkstyle 3m 31s trunk passed +1 mvnsite 20m 19s trunk passed +1 shadedclient 34m 1s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 34s trunk passed +1 javadoc 5m 42s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 27m 39s the patch passed +1 compile 28m 28s the patch passed -1 cc 28m 28s root generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11) +1 javac 28m 28s the patch passed -0 checkstyle 3m 35s root: The patch generated 17 new + 212 unchanged - 1 fixed = 229 total (was 213) +1 mvnsite 24m 0s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 11m 10s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . -1 findbugs 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 javadoc 6m 53s the patch passed       Other Tests -1 unit 16m 14s root in the patch failed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 237m 48s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   Inconsistent synchronization of org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.stopped; locked 66% of time Unsynchronized access at CGroupElasticMemoryController.java:66% of time Unsynchronized access at CGroupElasticMemoryController.java: [line 242]   Nullcheck of conf at line 360 of value previously dereferenced in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.getOOMListenerExecutablePath(Configuration) At CGroupElasticMemoryController.java:360 of value previously dereferenced in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.getOOMListenerExecutablePath(Configuration) At CGroupElasticMemoryController.java: [line 357]   Exceptional return value of java.util.concurrent.ExecutorService.submit(Callable) ignored in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.run() At CGroupElasticMemoryController.java:ignored in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController.run() At CGroupElasticMemoryController.java: [line 222]   Self assignment of field CGroupElasticMemoryController.oomHandler in new org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController(Configuration, Context, CGroupsHandler, boolean, boolean, long) At CGroupElasticMemoryController.java:in new org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController(Configuration, Context, CGroupsHandler, boolean, boolean, long) At CGroupElasticMemoryController.java: [line 120] Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12921279/YARN-4599.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux a1f15c6c07fb 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / a3b416f maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 cc https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/diff-compile-cc-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/20587/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20587/testReport/ Max. process+thread count 1496 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20587/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 24s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 19s Maven dependency ordering for branch
            +1 mvninstall 23m 27s trunk passed
            +1 compile 27m 0s trunk passed
            +1 checkstyle 3m 25s trunk passed
            +1 mvnsite 18m 37s trunk passed
            +1 shadedclient 30m 53s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 38s trunk passed
            +1 javadoc 5m 37s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            -1 mvninstall 0m 29s hadoop-yarn-api in the patch failed.
            -1 mvninstall 0m 34s hadoop-yarn-common in the patch failed.
            -1 mvninstall 0m 18s hadoop-yarn-server-nodemanager in the patch failed.
            +1 compile 25m 34s the patch passed
            +1 cc 25m 34s the patch passed
            +1 javac 25m 34s the patch passed
            +1 checkstyle 2m 42s root: The patch generated 0 new + 211 unchanged - 1 fixed = 211 total (was 212)
            +1 mvnsite 20m 4s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 8m 59s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 41s the patch passed
            +1 javadoc 5m 0s the patch passed
                  Other Tests
            -1 unit 150m 3s root in the patch failed.
            +1 asflicense 0m 30s The patch does not generate ASF License warnings.
            333m 54s



            Reason Tests
            Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
              hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
              hadoop.hdfs.web.TestWebHdfsTimeouts
              hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
              hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
              hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922325/YARN-4599.001.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 536e9121f4c4 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 696a4be
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/20617/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
            mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/20617/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
            mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/20617/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20617/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20617/testReport/
            Max. process+thread count 3513 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20617/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 23m 27s trunk passed +1 compile 27m 0s trunk passed +1 checkstyle 3m 25s trunk passed +1 mvnsite 18m 37s trunk passed +1 shadedclient 30m 53s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 38s trunk passed +1 javadoc 5m 37s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch -1 mvninstall 0m 29s hadoop-yarn-api in the patch failed. -1 mvninstall 0m 34s hadoop-yarn-common in the patch failed. -1 mvninstall 0m 18s hadoop-yarn-server-nodemanager in the patch failed. +1 compile 25m 34s the patch passed +1 cc 25m 34s the patch passed +1 javac 25m 34s the patch passed +1 checkstyle 2m 42s root: The patch generated 0 new + 211 unchanged - 1 fixed = 211 total (was 212) +1 mvnsite 20m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 8m 59s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 41s the patch passed +1 javadoc 5m 0s the patch passed       Other Tests -1 unit 150m 3s root in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 333m 54s Reason Tests Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922325/YARN-4599.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 536e9121f4c4 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 696a4be maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/20617/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/20617/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/20617/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/20617/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20617/testReport/ Max. process+thread count 3513 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20617/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            The build issue seems to be unrelated. Restarting the build.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - The build issue seems to be unrelated. Restarting the build.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 18s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  trunk Compile Tests
            0 mvndep 1m 39s Maven dependency ordering for branch
            +1 mvninstall 23m 41s trunk passed
            +1 compile 26m 47s trunk passed
            +1 checkstyle 2m 36s trunk passed
            +1 mvnsite 18m 6s trunk passed
            +1 shadedclient 30m 5s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 13s trunk passed
            +1 javadoc 5m 18s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 17s Maven dependency ordering for patch
            +1 mvninstall 25m 12s the patch passed
            +1 compile 25m 8s the patch passed
            +1 cc 25m 8s the patch passed
            +1 javac 25m 8s the patch passed
            +1 checkstyle 2m 37s root: The patch generated 0 new + 211 unchanged - 1 fixed = 211 total (was 212)
            +1 mvnsite 17m 49s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 9m 7s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 22s the patch passed
            +1 javadoc 5m 0s the patch passed
                  Other Tests
            -1 unit 93m 42s root in the patch failed.
            +1 asflicense 0m 30s The patch does not generate ASF License warnings.
            275m 9s



            Reason Tests
            Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
              hadoop.hdfs.TestDFSClientRetries
              hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
              hadoop.yarn.conf.TestYarnConfigurationFields



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922325/YARN-4599.001.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 1ed164439130 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / d72c1651
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20640/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20640/testReport/
            Max. process+thread count 4318 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20640/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 1m 39s Maven dependency ordering for branch +1 mvninstall 23m 41s trunk passed +1 compile 26m 47s trunk passed +1 checkstyle 2m 36s trunk passed +1 mvnsite 18m 6s trunk passed +1 shadedclient 30m 5s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 13s trunk passed +1 javadoc 5m 18s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 25m 12s the patch passed +1 compile 25m 8s the patch passed +1 cc 25m 8s the patch passed +1 javac 25m 8s the patch passed +1 checkstyle 2m 37s root: The patch generated 0 new + 211 unchanged - 1 fixed = 211 total (was 212) +1 mvnsite 17m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 9m 7s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 22s the patch passed +1 javadoc 5m 0s the patch passed       Other Tests -1 unit 93m 42s root in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 275m 9s Reason Tests Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.yarn.conf.TestYarnConfigurationFields Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922325/YARN-4599.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 1ed164439130 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / d72c1651 maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20640/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20640/testReport/ Max. process+thread count 4318 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20640/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            Fixing unit test.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Fixing unit test.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 31s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  trunk Compile Tests
            0 mvndep 1m 39s Maven dependency ordering for branch
            +1 mvninstall 26m 46s trunk passed
            +1 compile 29m 31s trunk passed
            +1 checkstyle 3m 18s trunk passed
            +1 mvnsite 19m 32s trunk passed
            +1 shadedclient 33m 9s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 25s trunk passed
            +1 javadoc 5m 29s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            +1 mvninstall 27m 23s the patch passed
            +1 compile 28m 2s the patch passed
            +1 cc 28m 2s the patch passed
            +1 javac 28m 2s the patch passed
            +1 checkstyle 3m 17s root: The patch generated 0 new + 211 unchanged - 1 fixed = 211 total (was 212)
            +1 mvnsite 19m 15s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 10m 10s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 48s the patch passed
            +1 javadoc 5m 20s the patch passed
                  Other Tests
            -1 unit 168m 44s root in the patch failed.
            +1 asflicense 0m 39s The patch does not generate ASF License warnings.
            369m 7s



            Reason Tests
            Failed junit tests hadoop.hdfs.client.impl.TestBlockReaderLocal
              hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922546/YARN-4599.002.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 017a01de9a9e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 69aac69
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20652/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20652/testReport/
            Max. process+thread count 3097 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20652/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 1m 39s Maven dependency ordering for branch +1 mvninstall 26m 46s trunk passed +1 compile 29m 31s trunk passed +1 checkstyle 3m 18s trunk passed +1 mvnsite 19m 32s trunk passed +1 shadedclient 33m 9s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 25s trunk passed +1 javadoc 5m 29s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 27m 23s the patch passed +1 compile 28m 2s the patch passed +1 cc 28m 2s the patch passed +1 javac 28m 2s the patch passed +1 checkstyle 3m 17s root: The patch generated 0 new + 211 unchanged - 1 fixed = 211 total (was 212) +1 mvnsite 19m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 10m 10s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 48s the patch passed +1 javadoc 5m 20s the patch passed       Other Tests -1 unit 168m 44s root in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 369m 7s Reason Tests Failed junit tests hadoop.hdfs.client.impl.TestBlockReaderLocal   hadoop.hdfs.server.namenode.TestNameNodeRespectsBindHostKeys Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922546/YARN-4599.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 017a01de9a9e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 69aac69 maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20652/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20652/testReport/ Max. process+thread count 3097 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20652/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            HDFS is not related to the patch.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - HDFS is not related to the patch.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 55s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
                  trunk Compile Tests
            0 mvndep 2m 12s Maven dependency ordering for branch
            +1 mvninstall 29m 50s trunk passed
            +1 compile 28m 3s trunk passed
            +1 checkstyle 3m 13s trunk passed
            +1 mvnsite 19m 9s trunk passed
            +1 shadedclient 32m 12s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 38s trunk passed
            +1 javadoc 5m 21s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 16s Maven dependency ordering for patch
            +1 mvninstall 27m 24s the patch passed
            +1 compile 28m 5s the patch passed
            +1 cc 28m 5s the patch passed
            +1 javac 28m 5s the patch passed
            +1 checkstyle 3m 15s root: The patch generated 0 new + 211 unchanged - 1 fixed = 211 total (was 212)
            +1 mvnsite 19m 33s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 10m 6s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 49s the patch passed
            +1 javadoc 5m 26s the patch passed
                  Other Tests
            -1 unit 183m 17s root in the patch failed.
            +1 asflicense 0m 37s The patch does not generate ASF License warnings.
            385m 39s



            Reason Tests
            Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
              hadoop.hdfs.TestReadStripedFileWithMissingBlocks
              hadoop.hdfs.TestDFSClientRetries
              hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
              hadoop.hdfs.TestDistributedFileSystem
              hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
              hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitorResourceChange



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922781/YARN-4599.003.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux f6d5cba5a8dd 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / cc0310a
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20672/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20672/testReport/
            Max. process+thread count 3140 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20672/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 55s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests 0 mvndep 2m 12s Maven dependency ordering for branch +1 mvninstall 29m 50s trunk passed +1 compile 28m 3s trunk passed +1 checkstyle 3m 13s trunk passed +1 mvnsite 19m 9s trunk passed +1 shadedclient 32m 12s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 38s trunk passed +1 javadoc 5m 21s trunk passed       Patch Compile Tests 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 27m 24s the patch passed +1 compile 28m 5s the patch passed +1 cc 28m 5s the patch passed +1 javac 28m 5s the patch passed +1 checkstyle 3m 15s root: The patch generated 0 new + 211 unchanged - 1 fixed = 211 total (was 212) +1 mvnsite 19m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 10m 6s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 49s the patch passed +1 javadoc 5m 26s the patch passed       Other Tests -1 unit 183m 17s root in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 385m 39s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.TestDistributedFileSystem   hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor   hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitorResourceChange Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922781/YARN-4599.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux f6d5cba5a8dd 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / cc0310a maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20672/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20672/testReport/ Max. process+thread count 3140 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20672/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            Fixing unit tests. asuresh, FYI this patch affects oversubscription.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Fixing unit tests. asuresh , FYI this patch affects oversubscription.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 31s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 17s Maven dependency ordering for branch
            +1 mvninstall 23m 5s trunk passed
            +1 compile 27m 30s trunk passed
            +1 checkstyle 3m 46s trunk passed
            +1 mvnsite 18m 0s trunk passed
            +1 shadedclient 31m 2s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 11s trunk passed
            +1 javadoc 5m 1s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            +1 mvninstall 25m 8s the patch passed
            +1 compile 26m 40s the patch passed
            +1 cc 26m 40s the patch passed
            +1 javac 26m 40s the patch passed
            +1 checkstyle 3m 22s root: The patch generated 0 new + 214 unchanged - 1 fixed = 214 total (was 215)
            +1 mvnsite 18m 22s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 9m 9s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 28s the patch passed
            +1 javadoc 4m 59s the patch passed
                  Other Tests
            -1 unit 201m 3s root in the patch failed.
            +1 asflicense 0m 39s The patch does not generate ASF License warnings.
            385m 12s



            Reason Tests
            Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
              hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
              hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
              hadoop.hdfs.web.TestWebHdfsTimeouts
              hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
              hadoop.hdfs.client.impl.TestBlockReaderLocal
              hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
              hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
              hadoop.hdfs.TestCrcCorruption
              hadoop.hdfs.server.namenode.ha.TestBootstrapStandby



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922884/YARN-4599.004.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 7c998c71c226 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 274eee3
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20689/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20689/testReport/
            Max. process+thread count 3433 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20689/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 23m 5s trunk passed +1 compile 27m 30s trunk passed +1 checkstyle 3m 46s trunk passed +1 mvnsite 18m 0s trunk passed +1 shadedclient 31m 2s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 11s trunk passed +1 javadoc 5m 1s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 25m 8s the patch passed +1 compile 26m 40s the patch passed +1 cc 26m 40s the patch passed +1 javac 26m 40s the patch passed +1 checkstyle 3m 22s root: The patch generated 0 new + 214 unchanged - 1 fixed = 214 total (was 215) +1 mvnsite 18m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 9m 9s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 28s the patch passed +1 javadoc 4m 59s the patch passed       Other Tests -1 unit 201m 3s root in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 385m 12s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.client.impl.TestBlockReaderLocal   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922884/YARN-4599.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 7c998c71c226 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 274eee3 maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20689/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20689/testReport/ Max. process+thread count 3433 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20689/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 26s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            0 mvndep 1m 36s Maven dependency ordering for branch
            +1 mvninstall 23m 9s trunk passed
            +1 compile 26m 41s trunk passed
            +1 checkstyle 3m 10s trunk passed
            +1 mvnsite 18m 37s trunk passed
            +1 shadedclient 30m 24s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 2m 59s trunk passed
            +1 javadoc 4m 59s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 15s Maven dependency ordering for patch
            +1 mvninstall 25m 2s the patch passed
            +1 compile 25m 50s the patch passed
            +1 cc 25m 50s the patch passed
            +1 javac 25m 50s the patch passed
            +1 checkstyle 2m 45s root: The patch generated 0 new + 214 unchanged - 1 fixed = 214 total (was 215)
            +1 mvnsite 18m 43s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 8m 50s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 38s the patch passed
            +1 javadoc 5m 6s the patch passed
                  Other Tests
            -1 unit 178m 30s root in the patch failed.
            +1 asflicense 0m 35s The patch does not generate ASF License warnings.
            360m 51s



            Reason Tests
            Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
              hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
              hadoop.hdfs.web.TestWebHdfsTimeouts
              hadoop.hdfs.client.impl.TestBlockReaderLocal
              hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922944/YARN-4599.005.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 8736fa58f91d 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 84b305f
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20695/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20695/testReport/
            Max. process+thread count 3649 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20695/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 1m 36s Maven dependency ordering for branch +1 mvninstall 23m 9s trunk passed +1 compile 26m 41s trunk passed +1 checkstyle 3m 10s trunk passed +1 mvnsite 18m 37s trunk passed +1 shadedclient 30m 24s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 2m 59s trunk passed +1 javadoc 4m 59s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 25m 2s the patch passed +1 compile 25m 50s the patch passed +1 cc 25m 50s the patch passed +1 javac 25m 50s the patch passed +1 checkstyle 2m 45s root: The patch generated 0 new + 214 unchanged - 1 fixed = 214 total (was 215) +1 mvnsite 18m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 8m 50s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 38s the patch passed +1 javadoc 5m 6s the patch passed       Other Tests -1 unit 178m 30s root in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 360m 51s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.client.impl.TestBlockReaderLocal   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12922944/YARN-4599.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 8736fa58f91d 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 84b305f maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20695/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20695/testReport/ Max. process+thread count 3649 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20695/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 32s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 20s Maven dependency ordering for branch
            +1 mvninstall 26m 56s trunk passed
            +1 compile 30m 49s trunk passed
            +1 checkstyle 3m 33s trunk passed
            +1 mvnsite 20m 35s trunk passed
            +1 shadedclient 34m 33s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 1s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 29s trunk passed
            +1 javadoc 5m 15s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            +1 mvninstall 27m 50s the patch passed
            +1 compile 28m 50s the patch passed
            +1 cc 28m 50s the patch passed
            +1 javac 28m 50s the patch passed
            +1 checkstyle 3m 58s root: The patch generated 0 new + 230 unchanged - 1 fixed = 230 total (was 231)
            +1 mvnsite 19m 51s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 10m 12s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            -1 findbugs 0m 25s hadoop-yarn-api in the patch failed.
            +1 javadoc 5m 23s the patch passed
                  Other Tests
            -1 unit 173m 16s root in the patch failed.
            +1 asflicense 0m 52s The patch does not generate ASF License warnings.
            376m 58s



            Reason Tests
            Failed junit tests hadoop.hdfs.client.impl.TestBlockReaderLocal
              hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
              hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923099/YARN-4599.006.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 5f8640f9ca02 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 4b4f24a
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20707/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20707/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20707/testReport/
            Max. process+thread count 3297 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20707/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 26m 56s trunk passed +1 compile 30m 49s trunk passed +1 checkstyle 3m 33s trunk passed +1 mvnsite 20m 35s trunk passed +1 shadedclient 34m 33s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 1s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 29s trunk passed +1 javadoc 5m 15s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 27m 50s the patch passed +1 compile 28m 50s the patch passed +1 cc 28m 50s the patch passed +1 javac 28m 50s the patch passed +1 checkstyle 3m 58s root: The patch generated 0 new + 230 unchanged - 1 fixed = 230 total (was 231) +1 mvnsite 19m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 10m 12s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . -1 findbugs 0m 25s hadoop-yarn-api in the patch failed. +1 javadoc 5m 23s the patch passed       Other Tests -1 unit 173m 16s root in the patch failed. +1 asflicense 0m 52s The patch does not generate ASF License warnings. 376m 58s Reason Tests Failed junit tests hadoop.hdfs.client.impl.TestBlockReaderLocal   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923099/YARN-4599.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 5f8640f9ca02 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 4b4f24a maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20707/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/20707/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20707/testReport/ Max. process+thread count 3297 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20707/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            haibochen Haibo Chen added a comment -

            Thanks miklos.szegedi@cloudera.com for the patch! The TestContainersMonitor.testContainerKillOnMemoryOverflow failure seems related.

            I have a few comments/questions:

            1) Error handling when writing to cgroup.event_control fails seems missing in oom_listener.c, do we need handle such an case?

            2) Is 'descriptors->event_control_fd = -1;'  necessary?

            3) The comments for test_oom() does not quite make sense to me. My current understanding is that it adds the calling process to the given pgroup and simulates an OOM by keep asking OS for memory?

            4) Can you please elaborate on how cgroup simulation is done in oom_listener_test_main.c? The child process that is added to the cgroup only does sleep().

            5) Doing a param matching in CGroupsHandlerImpl.GetCGroupParam() does not seem a good practice to me. Does it make sense to create a new method for the special case?

            6) Let's wrap the new thread join in ContainersMonitorImpl with try-catch clause as we do with the monitoring thread.

            7) The configuration changes are incompatible in that before the patch, poll-based pmcheck and vmcheck takes precedence over the cgroup based memory control mechanism. It is now reversed after the patch. if cgroup-based memory control is enabled, then poll-based pmcheck and vmcheck is disabled automatically. IIUC, one of the reasons is that we need to reuse the pmcheck and vmcheck flags which are dedicated to control the poll-based memory control.  How about we create separate configurations for pm elastic control and vm elastic control? We can make sure they are mutual exclusive as indicated in CGroupElasticMemoryController. We want to keep the elastic memory control mechanism independent of the per-container memory control mechanism, so we can get ride of the shortcut in checkLimit()  (warnings are probably more appropriate if we want to say the poll-based mechanism is not robust, which is an issue unrelated to what we are doing here.)

            8) In  CGroupElasticMemoryController, we can create a createOomHandler() method that is called by the constructor and overridden by the unit tests to avoid the test-only setOomHandler() method

            9) bq.       // TODO could we miss an event before the process starts?

            This is no longer an issue based on your experiment, per our offline discussion?

            10) We only need two threads in the thread pool, one for reading the error stream, and the other for watching and logging OOM state, don't we. If so, we change       executor = Executors.newFixedThreadPool(5);  =>       executor = Executors.newFixedThreadPool(2);

            11) I'm not quite sure how the watchdog thread can tell the elastic memory controller to stop. I guess once the watchdog thread calls stopListening(), the process is destroyed,  `(read = events.read(event)` would return false, we'd realize in the memory controller thread that OOM is not resolved in time and throw an exception to crash NM?  This process seems pretty obscure to me. Doe it make sense to have the stopListening logic in `if (!watchdog.get) {}` block instead?

            12) Can we replace thrown.expected() statements with @Test(expected) which is more declarative? Similarly in TestDefaultOOMHandler.

            13) In TestCGroupElasticMemoryController.testNormalExit(), not quite sure what the purpose of the sleep task is. Can you please add some comments there?

            14) Can we add some comments to DefaultOOMHandler javadoc, especially which containers are considered to be killed first.

            15) if new YarnRuntimeException("Could not find any containers but CGroups " + "reserved for containers ran out of memory. " + "I am giving up") is thrown in DefaultOOMHandler, CGroupElasticMemoryController  simply logs the exception. Do we want to crash the NM as well in this case?

            16) In TestDefaultOOMHandler.testBothContainersOOM(), I think we also need to verify container 2 is killed. Similarly, in  testOneContainerOOM() and  testNoContainerOOM().

             

             

            haibochen Haibo Chen added a comment - Thanks miklos.szegedi@cloudera.com for the patch! The TestContainersMonitor.testContainerKillOnMemoryOverflow failure seems related. I have a few comments/questions: 1) Error handling when writing to cgroup.event_control fails seems missing in oom_listener.c, do we need handle such an case? 2) Is 'descriptors->event_control_fd = -1;'  necessary? 3) The comments for test_oom() does not quite make sense to me. My current understanding is that it adds the calling process to the given pgroup and simulates an OOM by keep asking OS for memory? 4) Can you please elaborate on how cgroup simulation is done in oom_listener_test_main.c? The child process that is added to the cgroup only does sleep(). 5) Doing a param matching in CGroupsHandlerImpl.GetCGroupParam() does not seem a good practice to me. Does it make sense to create a new method for the special case? 6) Let's wrap the new thread join in ContainersMonitorImpl with try-catch clause as we do with the monitoring thread. 7) The configuration changes are incompatible in that before the patch, poll-based pmcheck and vmcheck takes precedence over the cgroup based memory control mechanism. It is now reversed after the patch. if cgroup-based memory control is enabled, then poll-based pmcheck and vmcheck is disabled automatically. IIUC, one of the reasons is that we need to reuse the pmcheck and vmcheck flags which are dedicated to control the poll-based memory control.  How about we create separate configurations for pm elastic control and vm elastic control? We can make sure they are mutual exclusive as indicated in CGroupElasticMemoryController. We want to keep the elastic memory control mechanism independent of the per-container memory control mechanism, so we can get ride of the shortcut in checkLimit()  (warnings are probably more appropriate if we want to say the poll-based mechanism is not robust, which is an issue unrelated to what we are doing here.) 8) In  CGroupElasticMemoryController, we can create a createOomHandler() method that is called by the constructor and overridden by the unit tests to avoid the test-only setOomHandler() method 9) bq.       // TODO could we miss an event before the process starts? This is no longer an issue based on your experiment, per our offline discussion? 10) We only need two threads in the thread pool, one for reading the error stream, and the other for watching and logging OOM state, don't we. If so, we change       executor = Executors.newFixedThreadPool(5);  =>       executor = Executors.newFixedThreadPool(2); 11) I'm not quite sure how the watchdog thread can tell the elastic memory controller to stop. I guess once the watchdog thread calls stopListening(), the process is destroyed,  `(read = events.read(event)` would return false, we'd realize in the memory controller thread that OOM is not resolved in time and throw an exception to crash NM?  This process seems pretty obscure to me. Doe it make sense to have the stopListening logic in `if (!watchdog.get) {}` block instead? 12) Can we replace thrown.expected() statements with @Test(expected) which is more declarative? Similarly in TestDefaultOOMHandler. 13) In TestCGroupElasticMemoryController.testNormalExit(), not quite sure what the purpose of the sleep task is. Can you please add some comments there? 14) Can we add some comments to DefaultOOMHandler javadoc, especially which containers are considered to be killed first. 15) if new YarnRuntimeException("Could not find any containers but CGroups " + "reserved for containers ran out of memory. " + "I am giving up") is thrown in DefaultOOMHandler, CGroupElasticMemoryController  simply logs the exception. Do we want to crash the NM as well in this case? 16) In TestDefaultOOMHandler.testBothContainersOOM(), I think we also need to verify container 2 is killed. Similarly, in  testOneContainerOOM() and  testNoContainerOOM().    
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

             Is 'descriptors->event_control_fd = -1;'  necessary?

            Yes it is a defense against chained errors, it may make it easier to debug when you get a core dump.

             3) The comments for test_oom() does not quite make sense to me. My current understanding is that it adds the calling process to the given pgroup and simulates an OOM by keep asking OS for memory?

            You are mixing the parent with the child. The parent gets the child pid and the child gets 0 after the fork() since the child can just call getpid(). It forks a child process gets it's pid in the parent and adds that to a cgroup. Once the child notices that it is in the cgroup it starts eating memory triggering an OOM.

             4) Can you please elaborate on how cgroup simulation is done in oom_listener_test_main.c? The child process that is added to the cgroup only does sleep().

            /*
            Unit test for cgroup testing. There are two modes.
            If the unit test is run as root and we have cgroups
            we try to crate a cgroup and generate an OOM.
            If we are not running as root we just sleep instead
            of eating memory and simulate the OOM by sending
            an event in a mock event fd mock_oom_event_as_user.
            */

             5) Doing a param matching in CGroupsHandlerImpl.GetCGroupParam() does not seem a good practice to me.

            CGroupsHandlerImpl.GetCGroupParam() is a smart function that returns the file name given the parameter name. I do not see any good practice issue here. The tasks file is always without the controller name.

             6) Let's wrap the new thread join in ContainersMonitorImpl with try-catch clause as we do with the monitoring thread.

            May I ask why? I thought only exceptions that will actually be thrown need to be caught. CGroupElasticMemoryController has a much better cleanup process than the monitoring thread and it does not need InterruptedException. In fact any interrupted exception would mean that we have likely leaked the external process, so I would advise against using it.

             7) The configuration changes are incompatible  ... How about we create separate configurations for pm elastic control and vm elastic control?

            I do not necessarily agree here.

            a) First of all polling and cgroups memory control did not work together before the patch either. NM exited with an exception, so there is no previous functionality that worked before and it does not work now. There is no compatibility break. cgroups takes a precedence indeed, that is a new feature.

            b) I would like to have a clean design in the long term for configuration avoiding too many configuration entries and definitely avoiding confusion. If here is a yarn.nodemanager.pmem-check-enabled, it suggests general use, it would be unintuitive not to use it. We indeed cannot change it's general meaning anymore. I think the clean design is having yarn.nodemanager.resource.memory.enabled to enable cgroups, yarn.nodemanager.resource.memory.enforced to enforce it per container and yarn.nodemanager.elastic-memory-control.enabled to enforce it at the node level. The detailed settings like yarn.nodemanager.pmem-check-enabled and yarn.nodemanager.pmem-check-enabled can only intuitively apply to all of them. In uderstand the concern but this solution would let us keep only these five configuration entries.

            11) Does it make sense to have the stopListening logic in `if (!watchdog.get) {}` block instead?

            It is completely equivalent. It will be called a few milliseconds earlier later, but there was a missing explanation there, so I added a comment.

             16) In TestDefaultOOMHandler.testBothContainersOOM(), I think we also need to verify container 2 is killed. Similarly, in  testOneContainerOOM() and  testNoContainerOOM().

            Only one container should be killed. However, I refined the verify logic to be even more precise verifying this.

            I addressed the rest. I will provide a patch soon.

             

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -  Is 'descriptors->event_control_fd = -1;'  necessary? Yes it is a defense against chained errors, it may make it easier to debug when you get a core dump.  3) The comments for test_oom() does not quite make sense to me. My current understanding is that it adds the calling process to the given pgroup and simulates an OOM by keep asking OS for memory? You are mixing the parent with the child. The parent gets the child pid and the child gets 0 after the fork() since the child can just call getpid(). It forks a child process gets it's pid in the parent and adds that to a cgroup. Once the child notices that it is in the cgroup it starts eating memory triggering an OOM.  4) Can you please elaborate on how cgroup simulation is done in oom_listener_test_main.c? The child process that is added to the cgroup only does sleep(). /* Unit test for cgroup testing. There are two modes. If the unit test is run as root and we have cgroups we try to crate a cgroup and generate an OOM. If we are not running as root we just sleep instead of eating memory and simulate the OOM by sending an event in a mock event fd mock_oom_event_as_user. */  5) Doing a param matching in CGroupsHandlerImpl.GetCGroupParam() does not seem a good practice to me. CGroupsHandlerImpl.GetCGroupParam() is a smart function that returns the file name given the parameter name. I do not see any good practice issue here. The tasks file is always without the controller name.  6) Let's wrap the new thread join in ContainersMonitorImpl with try-catch clause as we do with the monitoring thread. May I ask why? I thought only exceptions that will actually be thrown need to be caught. CGroupElasticMemoryController has a much better cleanup process than the monitoring thread and it does not need InterruptedException. In fact any interrupted exception would mean that we have likely leaked the external process, so I would advise against using it.  7) The configuration changes are incompatible  ... How about we create separate configurations for pm elastic control and vm elastic control? I do not necessarily agree here. a) First of all polling and cgroups memory control did not work together before the patch either. NM exited with an exception, so there is no previous functionality that worked before and it does not work now. There is no compatibility break. cgroups takes a precedence indeed, that is a new feature. b) I would like to have a clean design in the long term for configuration avoiding too many configuration entries and definitely avoiding confusion. If here is a yarn.nodemanager.pmem-check-enabled, it suggests general use, it would be unintuitive not to use it. We indeed cannot change it's general meaning anymore. I think the clean design is having yarn.nodemanager.resource.memory.enabled to enable cgroups, yarn.nodemanager.resource.memory.enforced to enforce it per container and yarn.nodemanager.elastic-memory-control.enabled to enforce it at the node level. The detailed settings like yarn.nodemanager.pmem-check-enabled and yarn.nodemanager.pmem-check-enabled can only intuitively apply to all of them. In uderstand the concern but this solution would let us keep only these five configuration entries. 11) Does it make sense to have the stopListening logic in `if (!watchdog.get) {}` block instead? It is completely equivalent. It will be called a few milliseconds earlier later, but there was a missing explanation there, so I added a comment.  16) In TestDefaultOOMHandler.testBothContainersOOM(), I think we also need to verify container 2 is killed. Similarly, in  testOneContainerOOM() and  testNoContainerOOM(). Only one container should be killed. However, I refined the verify logic to be even more precise verifying this. I addressed the rest. I will provide a patch soon.  
            snemeth Szilard Nemeth added a comment -

            Thanks miklos.szegedi@cloudera.com for the patch.

            Couple of comments, some of them are minor, some of them could be important:

             

            1. I think you have a typo for the property: yarn.nodemanager.elastic-memory-control.enabled in yarn-default.xml
            "if all the containers exceeded the a limit." --> should be changed to: "...exceeded the limit." (without an "a")

            2. I think time unit should be provided for the property yarn.nodemanager.elastic-memory-control.timeout in yarn-default.xml.
            From the description, I guess the value 5 means seconds from the code, but only from the description it's ambiguous

            3. CGroupElasticMemoryController:

            A.) In constructor: I would create a separate method for create and return the OOM handler instance as the constructor is very crowded.
            As I see it, the oomHandlerClass and the instantiation logic both could be extracted to the method and then you don't need the oomHandlerTemp variable at all.

            B.) In constructor:
            I think it's cleaner to use YarnConfiguration constants in string messages (log messages, exception message).
            E.g. LOG.warn("yarn.nodemanager.elastic-memory-control.enabled... --> You could use YarnConfiguration.NM_ELASTIC_MEMORY_CONTROL_ENABLED instead.
            If you search for "yarn.nodemanager." in this class it will give you all the results I'm talking about.

            C.) [Question] In constructor: If both controlPhysicalMemory and controlVirtualMemory is on, you only warn log a line.
            How the NM will work in these conditions, in other words, is it enough to just log a warning and continue NM's execution?

            D.) In getOOMListenerExecutablePath(): You print OOM listener's path on debug level with:

            conf.get(
             YarnConfiguration.NM_ELASTIC_MEMORY_CONTROL_OOM_LISTENER_PATH,
             defaultPath)

            I think you should extract this to a final local variable as you are returning the same value.

            E.) In run(): On the first line of this method, you have a typo in oomNotResoved --> should be oomNotResolved.

            F.) In run(): You have a TODO in this method. Do you have any plans to resolve it or it will be resolved later?

            G.) In run(): I'm curious whether it can happen that in the while loop's statement, events.read would read more data than 8 bytes or is it perfectly safe to rely on that on every read, at most 8 bytes will be read?

            H.) In run(): Since can harldy fit on one screen, I would extract the substantive part of the run() method to a new method, i.e. from the first executor.submit() call (errorListener) to the last statement until the catch.

            I.) In run(): throw new YarnRuntimeException("OOM was not resolved in time.");} --> I would include how much time was spent in the log message for better troubleshooting.

            J.) Javadoc of watchAndLogOOMState return is wrong, it returns a boolean and not an empty string.

            K.) In setCGroupParameters(): Maybe in the ResourceHandlerException's message would be more understandable with whole words, i.e. instead of p/v abbreviations, I would use physical/virtual.

            L.) In watchAndLogOOMState(): Please extract "under_oom 1" to a static final String. This string is also used in DefaultOOMHandler so you should use the same constant.

            4. DefaultOOMHandler

            A.) There are several TODOs in this class, what is the plan with those?
            A.) Constructor could be package-private
            B.) (minor) Javadoc of run() has a typo, should be: "...exceeded its limits" instead of "it's limits". There are several other typos like this in this class.
            B.) In run(): Call to

             

            ArrayList<Container> containers = new ArrayList<>();
             containers.addAll(context.getContainers().values());
            

            could be this one liner:

            ArrayList<Container> containers = new ArrayList<>(context.getContainers().values());

            C.) In killContainerIfOOM():
            This line specifically:

            long request = container.getResource().getMemorySize() * 1024 * 1024;

            According to javadoc of org.apache.hadoop.yarn.api.records.Resource#getMemorySize, this will return MB of memory size as a default and it depends on the unit of the resource.
            You multiply it by 1024 two times, so I guess you are relying on that memory size would be bytes instead.
            This one seems a possible unit conversion issue for me.

            5.) TestCGroupElasticMemoryController:

            A.) testConstructorHandler(): It's not clear for me why do you expect YarnException with this test.
            Moreover, YarnException could be thrown if the CGroupMemoryController's constructor could not create an instance of the OOM handler which could hide a code error. I would throw a special exception from DummyRunnable's constructor and assert on that exception from the testcase in order to be sure that the OOM handler's constructor is invoked without an issue.

            snemeth Szilard Nemeth added a comment - Thanks miklos.szegedi@cloudera.com for the patch. Couple of comments, some of them are minor, some of them could be important:   1. I think you have a typo for the property: yarn.nodemanager.elastic-memory-control.enabled in yarn-default.xml "if all the containers exceeded the a limit." --> should be changed to: "...exceeded the limit." (without an "a") 2. I think time unit should be provided for the property yarn.nodemanager.elastic-memory-control.timeout in yarn-default.xml. From the description, I guess the value 5 means seconds from the code, but only from the description it's ambiguous 3. CGroupElasticMemoryController : A.) In constructor: I would create a separate method for create and return the OOM handler instance as the constructor is very crowded. As I see it, the oomHandlerClass  and the instantiation logic both could be extracted to the method and then you don't need the oomHandlerTemp  variable at all. B.) In constructor: I think it's cleaner to use YarnConfiguration constants in string messages (log messages, exception message). E.g. LOG.warn("yarn.nodemanager.elastic-memory-control.enabled ... --> You could use YarnConfiguration.NM_ELASTIC_MEMORY_CONTROL_ENABLED  instead. If you search for "yarn.nodemanager." in this class it will give you all the results I'm talking about. C.) [Question]  In constructor: If both controlPhysicalMemory  and controlVirtualMemory  is on, you only warn log a line. How the NM will work in these conditions, in other words, is it enough to just log a warning and continue NM's execution? D.) In getOOMListenerExecutablePath() : You print OOM listener's path on debug level with: conf.get( YarnConfiguration.NM_ELASTIC_MEMORY_CONTROL_OOM_LISTENER_PATH, defaultPath) I think you should extract this to a final local variable as you are returning the same value. E.) In run(): On the first line of this method, you have a typo in oomNotResoved  --> should be oomNotResolved . F.) In run(): You have a TODO in this method. Do you have any plans to resolve it or it will be resolved later? G.) In run(): I'm curious whether it can happen that in the while loop's statement, events.read  would read more data than 8 bytes or is it perfectly safe to rely on that on every read, at most 8 bytes will be read? H.) In run(): Since can harldy fit on one screen, I would extract the substantive part of the run() method to a new method, i.e. from the first executor.submit()  call (errorListener) to the last statement until the catch. I.) In run(): throw new YarnRuntimeException("OOM was not resolved in time."); } --> I would include how much time was spent in the log message for better troubleshooting. J.) Javadoc of watchAndLogOOMState  return is wrong, it returns a boolean and not an empty string. K.) In setCGroupParameters() : Maybe in the ResourceHandlerException 's message would be more understandable with whole words, i.e. instead of p/v abbreviations, I would use physical/virtual. L.) In watchAndLogOOMState() : Please extract "under_oom 1"  to a static final String. This string is also used in DefaultOOMHandler  so you should use the same constant. 4. DefaultOOMHandler A.) There are several TODOs in this class, what is the plan with those? A.) Constructor could be package-private B.) (minor) Javadoc of run() has a typo, should be: "...exceeded its limits" instead of "it's limits". There are several other typos like this in this class. B.) In run(): Call to   ArrayList<Container> containers = new ArrayList<>(); containers.addAll(context.getContainers().values()); could be this one liner: ArrayList<Container> containers = new ArrayList<>(context.getContainers().values()); C.) In killContainerIfOOM() : This line specifically: long request = container.getResource().getMemorySize() * 1024 * 1024; According to javadoc of org.apache.hadoop.yarn.api.records.Resource#getMemorySize , this will return MB of memory size as a default and it depends on the unit of the resource. You multiply it by 1024 two times, so I guess you are relying on that memory size would be bytes instead. This one seems a possible unit conversion issue for me. 5.) TestCGroupElasticMemoryController : A.) testConstructorHandler() : It's not clear for me why do you expect YarnException  with this test. Moreover, YarnException  could be thrown if the CGroupMemoryController 's constructor could not create an instance of the OOM handler which could hide a code error. I would throw a special exception from DummyRunnable 's constructor and assert on that exception from the testcase in order to be sure that the OOM handler's constructor is invoked without an issue.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - - edited

            Thanks for the review snemeth.

             [Question] In constructor: If both controlPhysicalMemory and controlVirtualMemory is on, you only warn log a line.

            this.controlPhysicalMemory = controlPhysicalMemory && !controlVirtualMemory; virtual will override

            G.) In run(): I'm curious whether it can happen that in the while loop's statement, events.read would read more data than 8 bytes or is it perfectly safe to rely on that on every read, at most 8 bytes will be read?

            It cannot return more than the buffer size that is 8.

            H) I like to have big logic together.

             

            I.) In run(): throw new YarnRuntimeException("OOM was not resolved in time.");} --> I would include how much time was spent in the log message for better troubleshooting.

            The watchdog function handles this.

            B.) In run(): Call to...

            It does not matter, it wraps into two lines anyways.

            C.) In killContainerIfOOM()

            Not sure what you mean here. Yes there is a conversion, since the request is in MB and the limit is in bytes.

            A.) testConstructorHandler()

            Good catch.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - - edited Thanks for the review snemeth .   [Question]  In constructor: If both  controlPhysicalMemory  and  controlVirtualMemory  is on, you only warn log a line. this.controlPhysicalMemory = controlPhysicalMemory && !controlVirtualMemory; virtual will override G.) In run(): I'm curious whether it can happen that in the while loop's statement,  events.read  would read more data than 8 bytes or is it perfectly safe to rely on that on every read, at most 8 bytes will be read? It cannot return more than the buffer size that is 8. H) I like to have big logic together.   I.) In run():  throw new YarnRuntimeException("OOM was not resolved in time."); } --> I would include how much time was spent in the log message for better troubleshooting. The watchdog function handles this. B.) In run(): Call to... It does not matter, it wraps into two lines anyways. C.) In  killContainerIfOOM() :  Not sure what you mean here. Yes there is a conversion, since the request is in MB and the limit is in bytes. A.)  testConstructorHandler() Good catch.
            haibochen Haibo Chen added a comment -

            Thanks miklos.szegedi@cloudera.com for updating the patch! The new documentation, how to configure is very clear and easy to understand to me. A few follow-up comments.

            You are mixing the parent with the child. The parent gets the child pid and the child gets 0 after the fork() since the child can just call getpid(). It forks a child process gets it's pid in the parent and adds that to a cgroup. Once the child notices that it is in the cgroup it starts eating memory triggering an OOM

            Reading the code once more, test_oom in oom_listener_main.c happens to have the same name as oom_listener_test_main.c. The test_oom() function in oom_listener_main.c seems to be a helper function that simulates OOM situation. Let's add more comments to it so that it is clearly indicated.

            The DefaultOOMHandler checks both physical and virtual memory regardless of whether we'd want to just enforce physical or virtural memory. We can make a flag and checks one only dependent on whether physical or virtual memory is enabled.

            The  ContainerSignalContext.equals() is just for the unit test. We could either complete it by including more fields, or we can use Argument Matcher in TestDefaultOOMHandler.java as an alternative.

            The way that the watchdog in the elastic controller can stop the main controller thread from listening is not quite straightforward. How about we add a comment before the while(read()) loop to say read() returns -1 if the process is destroye?

            haibochen Haibo Chen added a comment - Thanks miklos.szegedi@cloudera.com for updating the patch! The new documentation, how to configure is very clear and easy to understand to me. A few follow-up comments. You are mixing the parent with the child. The parent gets the child pid and the child gets 0 after the fork() since the child can just call getpid(). It forks a child process gets it's pid in the parent and adds that to a cgroup. Once the child notices that it is in the cgroup it starts eating memory triggering an OOM Reading the code once more, test_oom in oom_listener_main.c happens to have the same name as oom_listener_test_main.c. The test_oom() function in oom_listener_main.c seems to be a helper function that simulates OOM situation. Let's add more comments to it so that it is clearly indicated. The DefaultOOMHandler checks both physical and virtual memory regardless of whether we'd want to just enforce physical or virtural memory. We can make a flag and checks one only dependent on whether physical or virtual memory is enabled. The  ContainerSignalContext.equals() is just for the unit test. We could either complete it by including more fields, or we can use Argument Matcher in TestDefaultOOMHandler.java as an alternative. The way that the watchdog in the elastic controller can stop the main controller thread from listening is not quite straightforward. How about we add a comment before the while(read()) loop to say read() returns -1 if the process is destroye?
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 34s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 20s Maven dependency ordering for branch
            +1 mvninstall 25m 36s trunk passed
            +1 compile 29m 13s trunk passed
            +1 checkstyle 3m 17s trunk passed
            +1 mvnsite 19m 59s trunk passed
            +1 shadedclient 33m 44s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 39s trunk passed
            +1 javadoc 5m 28s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 17s Maven dependency ordering for patch
            +1 mvninstall 29m 2s the patch passed
            +1 compile 29m 37s the patch passed
            +1 cc 29m 37s the patch passed
            +1 javac 29m 37s the patch passed
            -0 checkstyle 3m 18s root: The patch generated 13 new + 235 unchanged - 1 fixed = 248 total (was 236)
            +1 mvnsite 19m 36s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 10m 19s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            -1 findbugs 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
            +1 javadoc 5m 14s the patch passed
                  Other Tests
            -1 unit 172m 10s root in the patch failed.
            +1 asflicense 0m 44s The patch does not generate ASF License warnings.
            374m 17s



            Reason Tests
            FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
              Should org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController$OOMNotResolvedException be a static inner class? At CGroupElasticMemoryController.java:inner class? At CGroupElasticMemoryController.java:[lines 180-182]
              org.apache.hadoop.yarn.server.nodemanager.executor.ContainerSignalContext defines equals and uses Object.hashCode() At ContainerSignalContext.java:Object.hashCode() At ContainerSignalContext.java:[lines 104-110]
            Failed junit tests hadoop.hdfs.qjournal.server.TestJournalNodeSync
              hadoop.hdfs.server.namenode.ha.TestHAAppend



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923544/YARN-4599.007.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux ef95742ebf9b 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 2d6195c
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20733/artifact/out/diff-checkstyle-root.txt
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20733/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20733/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20733/testReport/
            Max. process+thread count 3260 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20733/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 25m 36s trunk passed +1 compile 29m 13s trunk passed +1 checkstyle 3m 17s trunk passed +1 mvnsite 19m 59s trunk passed +1 shadedclient 33m 44s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 39s trunk passed +1 javadoc 5m 28s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 29m 2s the patch passed +1 compile 29m 37s the patch passed +1 cc 29m 37s the patch passed +1 javac 29m 37s the patch passed -0 checkstyle 3m 18s root: The patch generated 13 new + 235 unchanged - 1 fixed = 248 total (was 236) +1 mvnsite 19m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 10m 19s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . -1 findbugs 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 5m 14s the patch passed       Other Tests -1 unit 172m 10s root in the patch failed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 374m 17s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   Should org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupElasticMemoryController$OOMNotResolvedException be a static inner class? At CGroupElasticMemoryController.java:inner class? At CGroupElasticMemoryController.java: [lines 180-182]   org.apache.hadoop.yarn.server.nodemanager.executor.ContainerSignalContext defines equals and uses Object.hashCode() At ContainerSignalContext.java:Object.hashCode() At ContainerSignalContext.java: [lines 104-110] Failed junit tests hadoop.hdfs.qjournal.server.TestJournalNodeSync   hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923544/YARN-4599.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux ef95742ebf9b 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 2d6195c maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20733/artifact/out/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20733/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/20733/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20733/testReport/ Max. process+thread count 3260 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20733/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            snemeth Szilard Nemeth added a comment - - edited

            Thanks miklos.szegedi@cloudera.com for the updated patch!

            LGTM, I just noticed some small things:

            1. CGroupElasticMemoryController constructor: oomHandlerTemp is unnecessary, please directly assign getDefaultOOMHandler's return value to field oomHandler.

            2. CGroupElasticMemoryController constructor: In the first and second LOG.warn() I guess you wanted to use NM_ELASTIC_MEMORY_CONTROL_ENABLED instead of NM_ELASTIC_MEMORY_CONTROL_OOM_TIMEOUT_SEC.

            3. DefaultOOMHandler.killContainerIfOOM: Sorry my comment was wrong, it's clear for me that it converts MB to bytes correctly.
            Maybe it's worth to consider using org.apache.hadoop.yarn.util.UnitsConversionUtil#convert as this is the main converter method used to convert values like this.

            4. DefaultOOMHandler constructor: typo in javadoc, "hander"

             

            Please fix the 2 checkstyle issues.

             

            Thanks!

            snemeth Szilard Nemeth added a comment - - edited Thanks miklos.szegedi@cloudera.com for the updated patch! LGTM, I just noticed some small things: 1. CGroupElasticMemoryController  constructor: oomHandlerTemp is unnecessary, please directly assign getDefaultOOMHandler 's return value to field oomHandler. 2. CGroupElasticMemoryController  constructor: In the first and second LOG.warn()  I guess you wanted to use NM_ELASTIC_MEMORY_CONTROL_ENABLED  instead of NM_ELASTIC_MEMORY_CONTROL_OOM_TIMEOUT_SEC . 3. DefaultOOMHandler.killContainerIfOOM : Sorry my comment was wrong, it's clear for me that it converts MB to bytes correctly. Maybe it's worth to consider using org.apache.hadoop.yarn.util.UnitsConversionUtil#convert  as this is the main converter method used to convert values like this. 4. DefaultOOMHandler  constructor: typo in javadoc, "hander"   Please fix the 2 checkstyle issues.   Thanks!
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 34s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 1m 41s Maven dependency ordering for branch
            +1 mvninstall 27m 30s trunk passed
            +1 compile 30m 9s trunk passed
            +1 checkstyle 3m 13s trunk passed
            +1 mvnsite 25m 14s trunk passed
            +1 shadedclient 40m 38s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 4m 33s trunk passed
            +1 javadoc 6m 59s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 23s Maven dependency ordering for patch
            +1 mvninstall 34m 36s the patch passed
            +1 compile 37m 3s the patch passed
            +1 cc 37m 3s the patch passed
            +1 javac 37m 3s the patch passed
            -0 checkstyle 3m 48s root: The patch generated 2 new + 235 unchanged - 1 fixed = 237 total (was 236)
            +1 mvnsite 24m 13s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 10m 22s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 4m 15s the patch passed
            +1 javadoc 5m 19s the patch passed
                  Other Tests
            -1 unit 171m 43s root in the patch failed.
            +1 asflicense 0m 40s The patch does not generate ASF License warnings.
            406m 20s



            Reason Tests
            Failed junit tests hadoop.hdfs.server.namenode.TestReencryptionWithKMS
              hadoop.hdfs.server.namenode.TestDecommissioningStatus
              hadoop.hdfs.server.balancer.TestBalancer
              hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923581/YARN-4599.009.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 32a8753c2303 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 8d3b39d
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20742/artifact/out/diff-checkstyle-root.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20742/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20742/testReport/
            Max. process+thread count 3067 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20742/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 1m 41s Maven dependency ordering for branch +1 mvninstall 27m 30s trunk passed +1 compile 30m 9s trunk passed +1 checkstyle 3m 13s trunk passed +1 mvnsite 25m 14s trunk passed +1 shadedclient 40m 38s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 4m 33s trunk passed +1 javadoc 6m 59s trunk passed       Patch Compile Tests 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 34m 36s the patch passed +1 compile 37m 3s the patch passed +1 cc 37m 3s the patch passed +1 javac 37m 3s the patch passed -0 checkstyle 3m 48s root: The patch generated 2 new + 235 unchanged - 1 fixed = 237 total (was 236) +1 mvnsite 24m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 10m 22s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 4m 15s the patch passed +1 javadoc 5m 19s the patch passed       Other Tests -1 unit 171m 43s root in the patch failed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 406m 20s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestReencryptionWithKMS   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923581/YARN-4599.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 32a8753c2303 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 8d3b39d maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20742/artifact/out/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/20742/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20742/testReport/ Max. process+thread count 3067 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20742/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            The HDFS test failures are unrelated.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - The HDFS test failures are unrelated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            Fixing 2 checkstyle issues.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Fixing 2 checkstyle issues.
            haibochen Haibo Chen added a comment -

            +1 on the latest patch, pending one minor comments. 

            In  getDefaultOOMHandler(), the flag that stipulates whether DefaultOOMHandler is supposed to check virtual memory is hardcoded to false. This should be dependent on controlPhysicalMemory or controlVirtualMemory.  We can move the assignment of the oomHandler after controlPhysicalMemory and controlVirtualMemory is initialized. Hence I'd also echo Szilard's previous comment, we can combine

            " Runnable oomHandlerTemp =getDefaultOOMHandler(conf, context, oomHandlerOverride);     this.oomHandler = oomHandlerTemp"

            with this.oomHandler = getDefaultOOMHandler(conf, context, oomHandlerOverride, controlVirtualMemory);

            haibochen Haibo Chen added a comment - +1 on the latest patch, pending one minor comments.  In  getDefaultOOMHandler(), the flag that stipulates whether DefaultOOMHandler is supposed to check virtual memory is hardcoded to false. This should be dependent on controlPhysicalMemory or controlVirtualMemory.  We can move the assignment of the oomHandler after controlPhysicalMemory and controlVirtualMemory is initialized. Hence I'd also echo Szilard's previous comment, we can combine " Runnable oomHandlerTemp =getDefaultOOMHandler(conf, context, oomHandlerOverride);     this.oomHandler = oomHandlerTemp" with this.oomHandler = getDefaultOOMHandler(conf, context, oomHandlerOverride, controlVirtualMemory);
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 43s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 23s Maven dependency ordering for branch
            +1 mvninstall 24m 35s trunk passed
            +1 compile 30m 6s trunk passed
            +1 checkstyle 3m 21s trunk passed
            -1 mvnsite 5m 28s root in trunk failed.
            +1 shadedclient 19m 10s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 41s trunk passed
            +1 javadoc 5m 33s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 19s Maven dependency ordering for patch
            +1 mvninstall 26m 4s the patch passed
            +1 compile 26m 3s the patch passed
            +1 cc 26m 3s the patch passed
            +1 javac 26m 3s the patch passed
            +1 checkstyle 2m 43s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
            +1 mvnsite 18m 32s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 9m 21s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 36s the patch passed
            +1 javadoc 4m 55s the patch passed
                  Other Tests
            -1 unit 13m 48s root in the patch failed.
            +1 asflicense 0m 24s The patch does not generate ASF License warnings.
            191m 43s



            Reason Tests
            Failed junit tests hadoop.util.TestDiskChecker
              hadoop.util.TestReadWriteDiskValidator



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923716/YARN-4599.010.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 92eecd9ecbaa 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / d47c09d
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/20752/artifact/out/branch-mvnsite-root.txt
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20752/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20752/testReport/
            Max. process+thread count 1629 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20752/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 43s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 24m 35s trunk passed +1 compile 30m 6s trunk passed +1 checkstyle 3m 21s trunk passed -1 mvnsite 5m 28s root in trunk failed. +1 shadedclient 19m 10s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 41s trunk passed +1 javadoc 5m 33s trunk passed       Patch Compile Tests 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 26m 4s the patch passed +1 compile 26m 3s the patch passed +1 cc 26m 3s the patch passed +1 javac 26m 3s the patch passed +1 checkstyle 2m 43s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 mvnsite 18m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 9m 21s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 36s the patch passed +1 javadoc 4m 55s the patch passed       Other Tests -1 unit 13m 48s root in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 191m 43s Reason Tests Failed junit tests hadoop.util.TestDiskChecker   hadoop.util.TestReadWriteDiskValidator Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923716/YARN-4599.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 92eecd9ecbaa 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / d47c09d maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/20752/artifact/out/branch-mvnsite-root.txt findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20752/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20752/testReport/ Max. process+thread count 1629 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20752/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            snemeth, thanks for the review.

            oomHandlerTemp is necessary since we may throw an exception after we set it. An external code optimized by JVM (setting a variable before the constructor is finished) may get a partial copy of this object in case of the exception. It is better to set the fields after the exception is not thrown.

            3) This code has to be very fast do the same conversion for 1000s of containers potentially, so I will keep the plain multiplication instead of calling int external code that deals with strings.

             

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - snemeth , thanks for the review. oomHandlerTemp is necessary since we may throw an exception after we set it. An external code optimized by JVM (setting a variable before the constructor is finished) may get a partial copy of this object in case of the exception. It is better to set the fields after the exception is not thrown. 3) This code has to be very fast do the same conversion for 1000s of containers potentially, so I will keep the plain multiplication instead of calling int external code that deals with strings.  
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 17s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for branch
            +1 mvninstall 23m 11s trunk passed
            -1 compile 31m 25s root in trunk failed.
            +1 checkstyle 2m 36s trunk passed
            +1 mvnsite 18m 37s trunk passed
            +1 shadedclient 30m 31s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 11s trunk passed
            +1 javadoc 5m 3s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            +1 mvninstall 24m 48s the patch passed
            +1 compile 27m 0s the patch passed
            -1 cc 27m 0s root generated 5 new + 6 unchanged - 0 fixed = 11 total (was 6)
            -1 javac 27m 0s root generated 188 new + 1277 unchanged - 0 fixed = 1465 total (was 1277)
            +1 checkstyle 2m 41s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
            +1 mvnsite 17m 20s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 8m 25s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 21s the patch passed
            +1 javadoc 4m 52s the patch passed
                  Other Tests
            -1 unit 150m 24s root in the patch failed.
            +1 asflicense 0m 39s The patch does not generate ASF License warnings.
            335m 9s



            Reason Tests
            Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
              hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
              hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923724/YARN-4599.011.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux a9c64fb5a10d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / d47c09d
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            compile https://builds.apache.org/job/PreCommit-YARN-Build/20753/artifact/out/branch-compile-root.txt
            findbugs v3.1.0-RC1
            cc https://builds.apache.org/job/PreCommit-YARN-Build/20753/artifact/out/diff-compile-cc-root.txt
            javac https://builds.apache.org/job/PreCommit-YARN-Build/20753/artifact/out/diff-compile-javac-root.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20753/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20753/testReport/
            Max. process+thread count 4117 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20753/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 23m 11s trunk passed -1 compile 31m 25s root in trunk failed. +1 checkstyle 2m 36s trunk passed +1 mvnsite 18m 37s trunk passed +1 shadedclient 30m 31s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 11s trunk passed +1 javadoc 5m 3s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 24m 48s the patch passed +1 compile 27m 0s the patch passed -1 cc 27m 0s root generated 5 new + 6 unchanged - 0 fixed = 11 total (was 6) -1 javac 27m 0s root generated 188 new + 1277 unchanged - 0 fixed = 1465 total (was 1277) +1 checkstyle 2m 41s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 mvnsite 17m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 8m 25s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 21s the patch passed +1 javadoc 4m 52s the patch passed       Other Tests -1 unit 150m 24s root in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 335m 9s Reason Tests Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923724/YARN-4599.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux a9c64fb5a10d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / d47c09d maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 compile https://builds.apache.org/job/PreCommit-YARN-Build/20753/artifact/out/branch-compile-root.txt findbugs v3.1.0-RC1 cc https://builds.apache.org/job/PreCommit-YARN-Build/20753/artifact/out/diff-compile-cc-root.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/20753/artifact/out/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/20753/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20753/testReport/ Max. process+thread count 4117 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20753/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 23s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 1m 41s Maven dependency ordering for branch
            +1 mvninstall 23m 30s trunk passed
            +1 compile 27m 25s trunk passed
            +1 checkstyle 2m 46s trunk passed
            +1 mvnsite 18m 21s trunk passed
            +1 shadedclient 30m 45s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 18s trunk passed
            +1 javadoc 5m 1s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 17s Maven dependency ordering for patch
            +1 mvninstall 24m 53s the patch passed
            +1 compile 26m 16s the patch passed
            +1 cc 26m 16s the patch passed
            +1 javac 26m 16s the patch passed
            -0 checkstyle 2m 35s root: The patch generated 1 new + 235 unchanged - 1 fixed = 236 total (was 236)
            +1 mvnsite 17m 57s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 8m 36s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 22s the patch passed
            +1 javadoc 4m 46s the patch passed
                  Other Tests
            -1 unit 14m 9s root in the patch failed.
            +1 asflicense 0m 31s The patch does not generate ASF License warnings.
            196m 46s



            Reason Tests
            Failed junit tests hadoop.util.TestBasicDiskValidator



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923821/YARN-4599.012.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 96c438297f44 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / be53969
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20761/artifact/out/diff-checkstyle-root.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20761/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20761/testReport/
            Max. process+thread count 1608 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20761/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 1m 41s Maven dependency ordering for branch +1 mvninstall 23m 30s trunk passed +1 compile 27m 25s trunk passed +1 checkstyle 2m 46s trunk passed +1 mvnsite 18m 21s trunk passed +1 shadedclient 30m 45s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 18s trunk passed +1 javadoc 5m 1s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 24m 53s the patch passed +1 compile 26m 16s the patch passed +1 cc 26m 16s the patch passed +1 javac 26m 16s the patch passed -0 checkstyle 2m 35s root: The patch generated 1 new + 235 unchanged - 1 fixed = 236 total (was 236) +1 mvnsite 17m 57s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 8m 36s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 22s the patch passed +1 javadoc 4m 46s the patch passed       Other Tests -1 unit 14m 9s root in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 196m 46s Reason Tests Failed junit tests hadoop.util.TestBasicDiskValidator Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923821/YARN-4599.012.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 96c438297f44 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / be53969 maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20761/artifact/out/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/20761/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20761/testReport/ Max. process+thread count 1608 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20761/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 32s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 19s Maven dependency ordering for branch
            +1 mvninstall 25m 54s trunk passed
            +1 compile 29m 43s trunk passed
            +1 checkstyle 3m 15s trunk passed
            +1 mvnsite 20m 13s trunk passed
            +1 shadedclient 33m 49s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 44s trunk passed
            +1 javadoc 6m 0s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 21s Maven dependency ordering for patch
            +1 mvninstall 29m 4s the patch passed
            +1 compile 28m 56s the patch passed
            +1 cc 28m 56s the patch passed
            +1 javac 28m 56s the patch passed
            +1 checkstyle 3m 25s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
            +1 mvnsite 19m 1s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 10m 11s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 51s the patch passed
            +1 javadoc 5m 14s the patch passed
                  Other Tests
            -1 unit 174m 38s root in the patch failed.
            +1 asflicense 0m 39s The patch does not generate ASF License warnings.
            377m 5s



            Reason Tests
            Failed junit tests hadoop.hdfs.TestPread



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923724/YARN-4599.011.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 10ba4bc48f60 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / be53969
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20755/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20755/testReport/
            Max. process+thread count 3092 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20755/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 25m 54s trunk passed +1 compile 29m 43s trunk passed +1 checkstyle 3m 15s trunk passed +1 mvnsite 20m 13s trunk passed +1 shadedclient 33m 49s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 44s trunk passed +1 javadoc 6m 0s trunk passed       Patch Compile Tests 0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 29m 4s the patch passed +1 compile 28m 56s the patch passed +1 cc 28m 56s the patch passed +1 javac 28m 56s the patch passed +1 checkstyle 3m 25s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 mvnsite 19m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 10m 11s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 51s the patch passed +1 javadoc 5m 14s the patch passed       Other Tests -1 unit 174m 38s root in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 377m 5s Reason Tests Failed junit tests hadoop.hdfs.TestPread Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923724/YARN-4599.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 10ba4bc48f60 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / be53969 maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20755/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20755/testReport/ Max. process+thread count 3092 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20755/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 32s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 1m 36s Maven dependency ordering for branch
            +1 mvninstall 23m 27s trunk passed
            +1 compile 27m 9s trunk passed
            +1 checkstyle 2m 48s trunk passed
            +1 mvnsite 18m 16s trunk passed
            +1 shadedclient 30m 29s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 12s trunk passed
            +1 javadoc 5m 9s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            +1 mvninstall 24m 56s the patch passed
            +1 compile 26m 11s the patch passed
            +1 cc 26m 11s the patch passed
            +1 javac 26m 11s the patch passed
            +1 checkstyle 3m 24s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
            +1 mvnsite 18m 1s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 8m 34s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 25s the patch passed
            +1 javadoc 4m 50s the patch passed
                  Other Tests
            -1 unit 14m 39s root in the patch failed.
            +1 asflicense 0m 25s The patch does not generate ASF License warnings.
            197m 42s



            Reason Tests
            Failed junit tests hadoop.util.TestBasicDiskValidator
              hadoop.util.TestDiskChecker
              hadoop.util.TestReadWriteDiskValidator



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923850/YARN-4599.013.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 684f59c2537c 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 454de3b
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20762/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20762/testReport/
            Max. process+thread count 1663 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20762/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 1m 36s Maven dependency ordering for branch +1 mvninstall 23m 27s trunk passed +1 compile 27m 9s trunk passed +1 checkstyle 2m 48s trunk passed +1 mvnsite 18m 16s trunk passed +1 shadedclient 30m 29s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 12s trunk passed +1 javadoc 5m 9s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 24m 56s the patch passed +1 compile 26m 11s the patch passed +1 cc 26m 11s the patch passed +1 javac 26m 11s the patch passed +1 checkstyle 3m 24s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 mvnsite 18m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 8m 34s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 25s the patch passed +1 javadoc 4m 50s the patch passed       Other Tests -1 unit 14m 39s root in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 197m 42s Reason Tests Failed junit tests hadoop.util.TestBasicDiskValidator   hadoop.util.TestDiskChecker   hadoop.util.TestReadWriteDiskValidator Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923850/YARN-4599.013.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 684f59c2537c 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 454de3b maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20762/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20762/testReport/ Max. process+thread count 1663 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20762/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 34s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 35s Maven dependency ordering for branch
            +1 mvninstall 25m 9s trunk passed
            +1 compile 28m 29s trunk passed
            +1 checkstyle 3m 16s trunk passed
            +1 mvnsite 19m 7s trunk passed
            +1 shadedclient 32m 40s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 23s trunk passed
            +1 javadoc 5m 15s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            +1 mvninstall 27m 15s the patch passed
            +1 compile 29m 1s the patch passed
            +1 cc 29m 1s the patch passed
            +1 javac 29m 1s the patch passed
            +1 checkstyle 3m 14s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
            +1 mvnsite 20m 45s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 10m 26s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 4m 5s the patch passed
            +1 javadoc 6m 16s the patch passed
                  Other Tests
            -1 unit 184m 34s root in the patch failed.
            +1 asflicense 0m 53s The patch does not generate ASF License warnings.
            384m 43s



            Reason Tests
            Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
              hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
              hadoop.hdfs.client.impl.TestBlockReaderLocal



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923852/YARN-4599.014.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 8d573ae47cba 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 454de3b
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20763/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20763/testReport/
            Max. process+thread count 3264 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20763/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 0m 35s Maven dependency ordering for branch +1 mvninstall 25m 9s trunk passed +1 compile 28m 29s trunk passed +1 checkstyle 3m 16s trunk passed +1 mvnsite 19m 7s trunk passed +1 shadedclient 32m 40s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 23s trunk passed +1 javadoc 5m 15s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 27m 15s the patch passed +1 compile 29m 1s the patch passed +1 cc 29m 1s the patch passed +1 javac 29m 1s the patch passed +1 checkstyle 3m 14s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 mvnsite 20m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 10m 26s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 4m 5s the patch passed +1 javadoc 6m 16s the patch passed       Other Tests -1 unit 184m 34s root in the patch failed. +1 asflicense 0m 53s The patch does not generate ASF License warnings. 384m 43s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.client.impl.TestBlockReaderLocal Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923852/YARN-4599.014.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 8d573ae47cba 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 454de3b maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20763/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20763/testReport/ Max. process+thread count 3264 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20763/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 32s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 19s Maven dependency ordering for branch
            +1 mvninstall 26m 13s trunk passed
            +1 compile 30m 38s trunk passed
            +1 checkstyle 3m 22s trunk passed
            +1 mvnsite 20m 55s trunk passed
            +1 shadedclient 35m 4s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 32s trunk passed
            +1 javadoc 6m 28s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 23s Maven dependency ordering for patch
            +1 mvninstall 32m 56s the patch passed
            +1 compile 36m 30s the patch passed
            +1 cc 36m 30s the patch passed
            +1 javac 36m 30s the patch passed
            +1 checkstyle 3m 51s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
            +1 mvnsite 26m 2s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 10m 20s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 54s the patch passed
            +1 javadoc 5m 17s the patch passed
                  Other Tests
            -1 unit 172m 53s root in the patch failed.
            +1 asflicense 0m 47s The patch does not generate ASF License warnings.
            397m 25s



            Reason Tests
            Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
              hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923863/YARN-4599.015.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 51f8864b6cad 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 454de3b
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20764/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20764/testReport/
            Max. process+thread count 3156 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20764/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 26m 13s trunk passed +1 compile 30m 38s trunk passed +1 checkstyle 3m 22s trunk passed +1 mvnsite 20m 55s trunk passed +1 shadedclient 35m 4s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 32s trunk passed +1 javadoc 6m 28s trunk passed       Patch Compile Tests 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 32m 56s the patch passed +1 compile 36m 30s the patch passed +1 cc 36m 30s the patch passed +1 javac 36m 30s the patch passed +1 checkstyle 3m 51s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 mvnsite 26m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 10m 20s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 54s the patch passed +1 javadoc 5m 17s the patch passed       Other Tests -1 unit 172m 53s root in the patch failed. +1 asflicense 0m 47s The patch does not generate ASF License warnings. 397m 25s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12923863/YARN-4599.015.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 51f8864b6cad 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 454de3b maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20764/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20764/testReport/ Max. process+thread count 3156 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20764/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            snemeth Szilard Nemeth added a comment -

            Thanks miklos.szegedi@cloudera.com for your answer and the updated patch!

            About your response to 3: fair enough then, the multiplication can be way faster in this sense.

            Checked your latest patch (015), LGTM (non-binding).

            snemeth Szilard Nemeth added a comment - Thanks miklos.szegedi@cloudera.com for your answer and the updated patch! About your response to 3: fair enough then, the multiplication can be way faster in this sense. Checked your latest patch (015), LGTM (non-binding).
            haibochen Haibo Chen added a comment -

            Two more comments:

            1) The oomHandler will still have the flag as false because controlVirtualMemory has not been initialized in  getDefaultOOMHandler().

            2)  Let's change  /** Specifies whether to do memory check on overall usage vs
               * individual containers. / => /** Specifies whether to do memory check on overall usage */

            haibochen Haibo Chen added a comment - Two more comments: 1) The oomHandler will still have the flag as false because controlVirtualMemory has not been initialized in  getDefaultOOMHandler(). 2)  Let's change  /** Specifies whether to do memory check on overall usage vs    * individual containers. / => / ** Specifies whether to do memory check on overall usage */
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 33s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 17s Maven dependency ordering for branch
            +1 mvninstall 27m 51s trunk passed
            +1 compile 30m 10s trunk passed
            +1 checkstyle 3m 16s trunk passed
            +1 mvnsite 19m 8s trunk passed
            +1 shadedclient 32m 53s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 3m 47s trunk passed
            +1 javadoc 5m 32s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 18s Maven dependency ordering for patch
            +1 mvninstall 28m 58s the patch passed
            +1 compile 29m 41s the patch passed
            +1 cc 29m 41s the patch passed
            +1 javac 29m 41s the patch passed
            +1 checkstyle 3m 18s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236)
            +1 mvnsite 20m 28s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 10m 9s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site .
            +1 findbugs 4m 18s the patch passed
            +1 javadoc 6m 2s the patch passed
                  Other Tests
            -1 unit 166m 35s root in the patch failed.
            +1 asflicense 0m 40s The patch does not generate ASF License warnings.
            373m 17s



            Reason Tests
            Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
              hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
              hadoop.hdfs.client.impl.TestBlockReaderLocal
              hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
              hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
              hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
              hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
              hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
              hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
              hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
              hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd
            JIRA Issue YARN-4599
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12924683/YARN-4599.016.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc
            uname Linux 60cb2f5795a6 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 5a91406
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20836/artifact/out/patch-unit-root.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20836/testReport/
            Max. process+thread count 3279 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: .
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20836/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.       trunk Compile Tests 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 27m 51s trunk passed +1 compile 30m 10s trunk passed +1 checkstyle 3m 16s trunk passed +1 mvnsite 19m 8s trunk passed +1 shadedclient 32m 53s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 3m 47s trunk passed +1 javadoc 5m 32s trunk passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 28m 58s the patch passed +1 compile 29m 41s the patch passed +1 cc 29m 41s the patch passed +1 javac 29m 41s the patch passed +1 checkstyle 3m 18s root: The patch generated 0 new + 235 unchanged - 1 fixed = 235 total (was 236) +1 mvnsite 20m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 10m 9s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . +1 findbugs 4m 18s the patch passed +1 javadoc 6m 2s the patch passed       Other Tests -1 unit 166m 35s root in the patch failed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 373m 17s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.client.impl.TestBlockReaderLocal   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain   hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd JIRA Issue YARN-4599 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12924683/YARN-4599.016.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc uname Linux 60cb2f5795a6 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 5a91406 maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/20836/artifact/out/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20836/testReport/ Max. process+thread count 3279 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/20836/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

            The unit test issues are not related to the patch.

            miklos.szegedi@cloudera.com Miklos Szegedi added a comment - The unit test issues are not related to the patch.
            haibochen Haibo Chen added a comment -

            +1 on the latest patch. Will check it in later today if no objections

            haibochen Haibo Chen added a comment - +1 on the latest patch. Will check it in later today if no objections
            haibochen Haibo Chen added a comment -

            Thanks sandflee for the initial proposal, miklos.szegedi@cloudera.com for the patch and everyone else for the discussion! I have now committed the patch to trunk!

            haibochen Haibo Chen added a comment - Thanks sandflee for the initial proposal, miklos.szegedi@cloudera.com for the patch and everyone else for the discussion! I have now committed the patch to trunk!
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14277 (See https://builds.apache.org/job/Hadoop-trunk-Commit/14277/)
            YARN-4599. Set OOM control for memory cgroups. (Miklos Szegedi via Haibo (haibochen: rev d9964799544eefcf424fcc178d987525f5356cdf)

            • (edit) .gitignore
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupElasticMemoryController.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/executor/ContainerSignalContext.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DummyRunnableWithContext.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener.c
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener_main.c
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsMemoryResourceHandlerImpl.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener.h
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupElasticMemoryController.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestDefaultOOMHandler.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCGroupsMemory.md
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DefaultOOMHandler.java
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14277 (See https://builds.apache.org/job/Hadoop-trunk-Commit/14277/ ) YARN-4599 . Set OOM control for memory cgroups. (Miklos Szegedi via Haibo (haibochen: rev d9964799544eefcf424fcc178d987525f5356cdf) (edit) .gitignore (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupElasticMemoryController.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/test/oom_listener_test_main.cc (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/executor/ContainerSignalContext.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DummyRunnableWithContext.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener.c (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener_main.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsMemoryResourceHandlerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/oom-listener/impl/oom_listener.h (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupElasticMemoryController.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestDefaultOOMHandler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerCGroupsMemory.md (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DefaultOOMHandler.java

            TestCGroupElasticMemoryController is failing precommit builds recently, see YARN-8375. haibochen miklos.szegedi@cloudera.com can you take a look?

            jlowe Jason Darrell Lowe added a comment - TestCGroupElasticMemoryController is failing precommit builds recently, see YARN-8375 . haibochen miklos.szegedi@cloudera.com can you take a look?
            szegedim Miklos Szegedi added a comment -

            jlowe, this should be addressed now.

            szegedim Miklos Szegedi added a comment - jlowe , this should be addressed now.

            People

              miklos.szegedi@cloudera.com Miklos Szegedi
              kasha Karthik Kambatla
              Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: