Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:

      Description

      In RHEL7, the CPU controller is named "cpu,cpuacct". The comma in the controller name leads to container launch failure.

      RHEL7 deprecates libcgroup and recommends the user of systemd. However, systemd has certain shortcomings as identified in this JIRA (see comments).

      This JIRA only fixes the failure, and doesn't try to use systemd.

      1. YARN-2194-1.patch
        13 kB
        Wei Yan
      2. YARN-2194-2.patch
        1 kB
        Wei Yan
      3. YARN-2194-3.patch
        2 kB
        Wei Yan
      4. YARN-2194-4.patch
        6 kB
        Wei Yan
      5. YARN-2194-5.patch
        12 kB
        Wei Yan
      6. YARN-2194-6.patch
        14 kB
        Wei Yan
      7. YARN-2194-7.patch
        1 kB
        Sidharta Seethana

        Issue Links

          Activity

          Hide
          jonbringhurst Jon Bringhurst added a comment -

          It might also be useful to have a "SystemdNspawnContainerExectuor" for yarn.nodemanager.container-executor.class. I don't know how many people would be interesting it using it however.

          Show
          jonbringhurst Jon Bringhurst added a comment - It might also be useful to have a "SystemdNspawnContainerExectuor" for yarn.nodemanager.container-executor.class. I don't know how many people would be interesting it using it however.
          Hide
          ywskycn Wei Yan added a comment -

          SystemdNspawnContainerExectuor is a good idea. Just add one for systemd besides the standard CgroupsLCEHandler.

          Show
          ywskycn Wei Yan added a comment - SystemdNspawnContainerExectuor is a good idea. Just add one for systemd besides the standard CgroupsLCEHandler.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          do we need to have a special container executor? Or a specialized LCEResourcesHandler would do the trick?

          Show
          tucu00 Alejandro Abdelnur added a comment - do we need to have a special container executor? Or a specialized LCEResourcesHandler would do the trick?
          Hide
          ywskycn Wei Yan added a comment -

          Alejandro Abdelnur, you mean just detect whether the system is RedHat 7 inside the CgroupsLCEResourceHandler? If so, use systemd to execute?

          Show
          ywskycn Wei Yan added a comment - Alejandro Abdelnur , you mean just detect whether the system is RedHat 7 inside the CgroupsLCEResourceHandler? If so, use systemd to execute?
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          i was not meaning autodetection, i was meaning that a new resource handler may be enough to deal with cgroups in RH7, without having to write a new LCE.

          Show
          tucu00 Alejandro Abdelnur added a comment - i was not meaning autodetection, i was meaning that a new resource handler may be enough to deal with cgroups in RH7, without having to write a new LCE.
          Hide
          ywskycn Wei Yan added a comment -

          oh, yes, adding a new LCEResourceHandler to handle the preExecute and afterExecute is enough.

          Show
          ywskycn Wei Yan added a comment - oh, yes, adding a new LCEResourceHandler to handle the preExecute and afterExecute is enough.
          Hide
          beckham007 Beckham007 added a comment -

          +1.
          A new LCEResourceHandler is needed. To support more resource isolation, we also need to have init(), preExecute() and postExecute() for different resource. Adding an abstract CgroupsResourceManager and its implement CPUResourceManager\MemResourceManager is good.

          Show
          beckham007 Beckham007 added a comment - +1. A new LCEResourceHandler is needed. To support more resource isolation, we also need to have init(), preExecute() and postExecute() for different resource. Adding an abstract CgroupsResourceManager and its implement CPUResourceManager\MemResourceManager is good.
          Hide
          ywskycn Wei Yan added a comment -

          A prelim patch that implements the systemd-based cpu resource isolation for Redhat 7.
          A summary:
          (1) Create a new resource handler SystemdLCEResourceHandler. Users can use this handle by configuring the field "yarn.nodemanager.linux-container-executor.resources-handler.class".
          (2) For each container, create one slice and one scope. The scope is put inside the slice, and cpuShare isolation is also attached to the slice. All container's slices are organized in a root slice (named "hadoop_yarn.slice" in default).

          Will add some testcases later.

          Show
          ywskycn Wei Yan added a comment - A prelim patch that implements the systemd-based cpu resource isolation for Redhat 7. A summary: (1) Create a new resource handler SystemdLCEResourceHandler. Users can use this handle by configuring the field "yarn.nodemanager.linux-container-executor.resources-handler.class". (2) For each container, create one slice and one scope. The scope is put inside the slice, and cpuShare isolation is also attached to the slice. All container's slices are organized in a root slice (named "hadoop_yarn.slice" in default). Will add some testcases later.
          Hide
          beckham007 Beckham007 added a comment -

          startSystemdSlice/stopSystemdSlice needs root privilege? Let container-executor to run "sudo systemctl start " ?

          Show
          beckham007 Beckham007 added a comment - startSystemdSlice/stopSystemdSlice needs root privilege? Let container-executor to run "sudo systemctl start " ?
          Hide
          ywskycn Wei Yan added a comment -

          Thanks your comments, Beckham007.

          startSystemdSlice/stopSystemdSlice needs root privilege?

          Yes, "systemctl start/stop slice" needs root privilege.

          Let container-executor to run "sudo systemctl start "?

          You mean adding start/stop slice function in the container-executor, and let SystemdLCEResourceHandler invokes these functions?

          Show
          ywskycn Wei Yan added a comment - Thanks your comments, Beckham007 . startSystemdSlice/stopSystemdSlice needs root privilege? Yes, "systemctl start/stop slice" needs root privilege. Let container-executor to run "sudo systemctl start "? You mean adding start/stop slice function in the container-executor, and let SystemdLCEResourceHandler invokes these functions?
          Hide
          bcwalrus bc Wong added a comment -

          container-executor.c

          • L1188: If initialize_user() fails, do you not need to cleanup?
          • L1194: Same for create_log_dirs(). Seems that goto cleanup is still warranted.
          • L1207: Missing space before S_IRWXU.
          • L1243: Nit. Hardcoding 55 here is error-prone. You could allocate a 4K buffer here, and use snprintf.
          • L1244: You need to check the return value from malloc(). Since you're running as root here, everything has to be extra careful.
          • L1255: On failure, would log the command being executed.
          Show
          bcwalrus bc Wong added a comment - container-executor.c L1188: If initialize_user() fails, do you not need to cleanup? L1194: Same for create_log_dirs(). Seems that goto cleanup is still warranted. L1207: Missing space before S_IRWXU. L1243: Nit. Hardcoding 55 here is error-prone. You could allocate a 4K buffer here, and use snprintf. L1244: You need to check the return value from malloc(). Since you're running as root here, everything has to be extra careful. L1255: On failure, would log the command being executed.
          Hide
          kasha Karthik Kambatla added a comment -

          container-executor.c - the new method significantly duplicates the existing one. Can we have separate methods to capture the differences and leave the original method as is.

          Show
          kasha Karthik Kambatla added a comment - container-executor.c - the new method significantly duplicates the existing one. Can we have separate methods to capture the differences and leave the original method as is.
          Hide
          ywskycn Wei Yan added a comment -

          Sure, will update a new patch by combing bc Wong comments.

          Show
          ywskycn Wei Yan added a comment - Sure, will update a new patch by combing bc Wong comments.
          Hide
          ywskycn Wei Yan added a comment -

          Sure, will update a new patch by combing bc Wong comments.

          Show
          ywskycn Wei Yan added a comment - Sure, will update a new patch by combing bc Wong comments.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Is there no API that we can use instead of spawning shells to set this up?

          We should have some auto-detection to chose the right plugin for the right OS.

          IAC, YARN-3443 changed the way resource isolation code is organized in the ResourceManager. YARN-3542 is migrating existing cgroups+cpu support to the new layout. We need to relook at this patch in light of those changes.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Is there no API that we can use instead of spawning shells to set this up? We should have some auto-detection to chose the right plugin for the right OS. IAC, YARN-3443 changed the way resource isolation code is organized in the ResourceManager. YARN-3542 is migrating existing cgroups+cpu support to the new layout. We need to relook at this patch in light of those changes.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Also, it will be really useful if we could find a way to support existing code on RHEL7, even if libcgroups is deprecated and the approach ends up becoming heavy-handed, perhaps with some manual steps. This page talks about ways of doing this.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Also, it will be really useful if we could find a way to support existing code on RHEL7, even if libcgroups is deprecated and the approach ends up becoming heavy-handed, perhaps with some manual steps. This page talks about ways of doing this.
          Hide
          ywskycn Wei Yan added a comment -

          Vinod Kumar Vavilapalli, discussed about this with Karthik offline. We thought it would be better just let the existing cgroups code work well in RedHat7. And after that, we can provide another systemd solution.
          I can update a patch later.

          Show
          ywskycn Wei Yan added a comment - Vinod Kumar Vavilapalli , discussed about this with Karthik offline. We thought it would be better just let the existing cgroups code work well in RedHat7. And after that, we can provide another systemd solution. I can update a patch later.
          Hide
          kasha Karthik Kambatla added a comment -

          RHEL7 recommends using systemd, but doesn't require us to do so. As Wei said, we thought it might be better to fix the immediate issue (which Wei claims is a simple fix) instead of switching over to use systemd. We could follow-up with a fix that uses systemd.

          Show
          kasha Karthik Kambatla added a comment - RHEL7 recommends using systemd, but doesn't require us to do so. As Wei said, we thought it might be better to fix the immediate issue (which Wei claims is a simple fix) instead of switching over to use systemd. We could follow-up with a fix that uses systemd.
          Hide
          ywskycn Wei Yan added a comment -

          In Redhat 7, the controller for cpu is "/sys/fs/cgroup/cpu,cpuacct". However, this patch cannot be supported by container-executor due to the comma. So the patch replaces it to "/sys/fs/cgroup/cpu". And "/sys/fs/cgroup/cpu" has already symlinked to "/sys/fs/cgroup/cpu,cpuacct".

          Show
          ywskycn Wei Yan added a comment - In Redhat 7, the controller for cpu is "/sys/fs/cgroup/cpu,cpuacct". However, this patch cannot be supported by container-executor due to the comma. So the patch replaces it to "/sys/fs/cgroup/cpu". And "/sys/fs/cgroup/cpu" has already symlinked to "/sys/fs/cgroup/cpu,cpuacct".
          Hide
          kasha Karthik Kambatla added a comment -

          Verified the patch works. Can we add more comments to say clarify why the patch replaces cpu,cpuacct with cpu? May be something along the lines of - "In RHEL7, the CPU controller is named 'cpu,cpuacct'. The comma in the controller name leads to container launch failure. Symlinks 'cpu' and 'cpuacct' point to 'cpu,cpuacct'. Using 'cpu' solves the issue."

          Show
          kasha Karthik Kambatla added a comment - Verified the patch works. Can we add more comments to say clarify why the patch replaces cpu,cpuacct with cpu? May be something along the lines of - "In RHEL7, the CPU controller is named 'cpu,cpuacct'. The comma in the controller name leads to container launch failure. Symlinks 'cpu' and 'cpuacct' point to 'cpu,cpuacct'. Using 'cpu' solves the issue."
          Hide
          kasha Karthik Kambatla added a comment -

          +1 otherwise.

          Vinod Kumar Vavilapalli, Alejandro Abdelnur - is this somewhat hacky approach reasonable?

          Show
          kasha Karthik Kambatla added a comment - +1 otherwise. Vinod Kumar Vavilapalli , Alejandro Abdelnur - is this somewhat hacky approach reasonable?
          Hide
          ywskycn Wei Yan added a comment -

          Thanks, Karthik Kambatla. Updated a patch adding more comments.

          Show
          ywskycn Wei Yan added a comment - Thanks, Karthik Kambatla . Updated a patch adding more comments.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Thinking out aloud, should we do OS specific checks for this?

          Also, does the newer CGroupsHandlerImpl also need to change? /cc Varun Vasudev.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Thinking out aloud, should we do OS specific checks for this? Also, does the newer CGroupsHandlerImpl also need to change? /cc Varun Vasudev .
          Hide
          mjacobs Matthew Jacobs added a comment -

          While this may work for the default RHEL7 configuration, this will break if someone happens to have mounted the same controllers to "/sys/fs/cgroup/cpuacct,cpu", or if the user mounted other controllers at the same path as well. What do you think about creating the symlink from "/sys/fs/cgroup/cpu" to the mounted path for cpu in all cases (unless it was actually mounted at /sys/fs/cgroup/cpu of course).

          Show
          mjacobs Matthew Jacobs added a comment - While this may work for the default RHEL7 configuration, this will break if someone happens to have mounted the same controllers to "/sys/fs/cgroup/cpuacct,cpu", or if the user mounted other controllers at the same path as well. What do you think about creating the symlink from "/sys/fs/cgroup/cpu" to the mounted path for cpu in all cases (unless it was actually mounted at /sys/fs/cgroup/cpu of course).
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Isn't it better to use a different separator that is less likely to be in use ( e.g ':' or '|' instead of ',' ) when invoking container-executor ? Granted that this is a (slightly) bigger change, but it seems like the right thing to do.

          Show
          sidharta-s Sidharta Seethana added a comment - Isn't it better to use a different separator that is less likely to be in use ( e.g ':' or '|' instead of ',' ) when invoking container-executor ? Granted that this is a (slightly) bigger change, but it seems like the right thing to do.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          To clarify, my comment was respect to this line in the description The comma in the controller name leads to container launch failure. . I believe switching separators or encoding arguments in some way is a better approach than requiring symlinks or transform "cpu,cpuacct" to "cpu" as the controller name.

          Show
          sidharta-s Sidharta Seethana added a comment - To clarify, my comment was respect to this line in the description The comma in the controller name leads to container launch failure. . I believe switching separators or encoding arguments in some way is a better approach than requiring symlinks or transform "cpu,cpuacct" to "cpu" as the controller name.
          Hide
          ywskycn Wei Yan added a comment -

          Sidharta Seethana, thanks for the advice. Use a different separator LGTM. In that way, we can trust the "cpu" controller, and can also help avoid doing OS-specific changes.
          Comments? Karthik Kambatla, Vinod Kumar Vavilapalli, Matthew Jacobs.

          And for the new CGroupsHandlerImpl, didn't find any problem when I checked the patch. Varun Vasudev, please correct me if I missed anything.

          Show
          ywskycn Wei Yan added a comment - Sidharta Seethana , thanks for the advice. Use a different separator LGTM. In that way, we can trust the "cpu" controller, and can also help avoid doing OS-specific changes. Comments? Karthik Kambatla , Vinod Kumar Vavilapalli , Matthew Jacobs . And for the new CGroupsHandlerImpl, didn't find any problem when I checked the patch. Varun Vasudev , please correct me if I missed anything.
          Hide
          kasha Karthik Kambatla added a comment -

          I haven't looked at it closely, but think Yarn doesn't pick the separator. If we could easily change the separator from within Yarn, that is without requiring any other environment changes by the admin, I ll be a +1 for that change. By the way, Linux allows anything but '/' and '%' for filenames. So, picking ':' or '|' is only less likely to cause issues in the future. Who would have thought they would use ', in a filename?

          If we continue with the patch posted here, I think Matthew Jacobs' suggestion makes sense.

          Show
          kasha Karthik Kambatla added a comment - I haven't looked at it closely, but think Yarn doesn't pick the separator. If we could easily change the separator from within Yarn, that is without requiring any other environment changes by the admin, I ll be a +1 for that change. By the way, Linux allows anything but '/' and '%' for filenames. So, picking ':' or '|' is only less likely to cause issues in the future. Who would have thought they would use ', in a filename? If we continue with the patch posted here, I think Matthew Jacobs ' suggestion makes sense.
          Hide
          mjacobs Matthew Jacobs added a comment -

          I'm confused, does this mean that you'll re-mount the cpu and cpuacct controllers? Do we know that other components in the RHEL7 world don't expect them to be in the default place?

          Show
          mjacobs Matthew Jacobs added a comment - I'm confused, does this mean that you'll re-mount the cpu and cpuacct controllers? Do we know that other components in the RHEL7 world don't expect them to be in the default place?
          Hide
          philipl Philip Langdale added a comment -

          You can remount controllers if you retain the same combination as the existing mount point, so I guess you could replace the ',' with something your parsing code can handle (or you could fix the parsing code). In general, life is a lot easier if you can avoid remounting as you then don't have to worry about managing their lifecycle.

          I'd argue the most robust thing to do is discover the existing mount point from /proc/mounts and then use it (assuming the comma parsing can be fixed) if it's present (and don't forget to respect the NodeManager's cgroup paths from /proc/self/mounts)

          Show
          philipl Philip Langdale added a comment - You can remount controllers if you retain the same combination as the existing mount point, so I guess you could replace the ',' with something your parsing code can handle (or you could fix the parsing code). In general, life is a lot easier if you can avoid remounting as you then don't have to worry about managing their lifecycle. I'd argue the most robust thing to do is discover the existing mount point from /proc/mounts and then use it (assuming the comma parsing can be fixed) if it's present (and don't forget to respect the NodeManager's cgroup paths from /proc/self/mounts)
          Hide
          sidharta-s Sidharta Seethana added a comment -

          There are two different issues here :

          • container-executor binary invocation uses ‘,’ as a separator when supplying a list of paths - which breaks when the path contains ‘,’
          • cpu,cpuacct are mounted together by default on RHEL7

          Now, for the latter issue : In CgroupsLCEResourcesHandler, the following steps occur :

          • If the yarn.nodemanager.linux-container-executor.cgroups.mount switch is enabled , the ‘cpu’ controller is explicitly mounted at the specified path.
          • (irrespective of the state of the switch) The /proc/mounts file (possibly updated by the previous step) is subsequently parsed to determine the mount locations for the various cgroup controllers - this parsing code seems to be correct even if cpu and cpuacct are mounted in one location.

          So, the thing we need to fix is the separator issue and we should be good. The important thing to remember is that there are two cgroups implementation classes ( CgroupsLCEResourcesHandler and CGroupsHandlerImpl ). Hopefully, this will be addressed soon ( YARN-3542 ) - or we risk divergence.

          Show
          sidharta-s Sidharta Seethana added a comment - There are two different issues here : container-executor binary invocation uses ‘,’ as a separator when supplying a list of paths - which breaks when the path contains ‘,’ cpu,cpuacct are mounted together by default on RHEL7 Now, for the latter issue : In CgroupsLCEResourcesHandler , the following steps occur : If the yarn.nodemanager.linux-container-executor.cgroups.mount switch is enabled , the ‘cpu’ controller is explicitly mounted at the specified path. (irrespective of the state of the switch) The /proc/mounts file (possibly updated by the previous step) is subsequently parsed to determine the mount locations for the various cgroup controllers - this parsing code seems to be correct even if cpu and cpuacct are mounted in one location. So, the thing we need to fix is the separator issue and we should be good. The important thing to remember is that there are two cgroups implementation classes ( CgroupsLCEResourcesHandler and CGroupsHandlerImpl ). Hopefully, this will be addressed soon ( YARN-3542 ) - or we risk divergence.
          Hide
          mjacobs Matthew Jacobs added a comment -

          Thanks, [sidharta-s]. So the change would be in how the container-executor accepts lists of paths, not attempting to re-mount the controllers, right? If I understand it correctly, that sounds like a good plan to me.

          Show
          mjacobs Matthew Jacobs added a comment - Thanks, [sidharta-s] . So the change would be in how the container-executor accepts lists of paths, not attempting to re-mount the controllers, right? If I understand it correctly, that sounds like a good plan to me.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Matthew Jacobs , Yes, that is what I am proposing. If we handle the path separation correctly, we should be able to continue using the current (deprecated, but still workable) mechanism for using cgroups.

          Show
          sidharta-s Sidharta Seethana added a comment - Matthew Jacobs , Yes, that is what I am proposing. If we handle the path separation correctly, we should be able to continue using the current (deprecated, but still workable) mechanism for using cgroups.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Hi Wei Yan,

          Would you be able to submit a patch with the requested changes? Maybe we should consider pulling this into 2.7.1 ?

          /cc Vinod Kumar Vavilapalli

          Show
          sidharta-s Sidharta Seethana added a comment - Hi Wei Yan , Would you be able to submit a patch with the requested changes? Maybe we should consider pulling this into 2.7.1 ? /cc Vinod Kumar Vavilapalli
          Hide
          ywskycn Wei Yan added a comment -

          Sidharta Seethana, yes, working on it.

          Show
          ywskycn Wei Yan added a comment - Sidharta Seethana , yes, working on it.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Wei Yan , thanks! Looking forward to your patch.

          Show
          sidharta-s Sidharta Seethana added a comment - Wei Yan , thanks! Looking forward to your patch.
          Hide
          ywskycn Wei Yan added a comment -

          Uploaded a patch by replacing comma with '%'.

          Show
          ywskycn Wei Yan added a comment - Uploaded a patch by replacing comma with '%'.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 56s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 37s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 36s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 6m 6s Tests passed in hadoop-yarn-server-nodemanager.
              43m 44s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738765/YARN-2194-4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6785661
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8232/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8232/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8232/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 36s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 6m 6s Tests passed in hadoop-yarn-server-nodemanager.     43m 44s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738765/YARN-2194-4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6785661 hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8232/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8232/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8232/console This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          I tried running jobs with the patch posted here, and ran into issues during localization:

          Localizer failed
          java.io.IOException: Application application_1434091083696_0001 initialization failed (exitCode=20) with output: main : command provided 0
          main : user is nobody
          main : requested yarn user is systest
          Failed to create directory /data/yarn/nm%/data1/yarn/nm/usercache/systest - No such file or directory
          
          	at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:241)
          	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1132)
          Caused by: ExitCodeException exitCode=20: 
          	at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
          	at org.apache.hadoop.util.Shell.run(Shell.java:455)
          	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
          	at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:232)
          	... 1 more
          
          Show
          kasha Karthik Kambatla added a comment - I tried running jobs with the patch posted here, and ran into issues during localization: Localizer failed java.io.IOException: Application application_1434091083696_0001 initialization failed (exitCode=20) with output: main : command provided 0 main : user is nobody main : requested yarn user is systest Failed to create directory /data/yarn/nm%/data1/yarn/nm/usercache/systest - No such file or directory at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:241) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1132) Caused by: ExitCodeException exitCode=20: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:232) ... 1 more
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Wei Yan , you'll need to change PrivilegedOperationExecutor as well

                if (noneArgsOnly == false) {
                  //We have already appended at least one tasks file.
                  finalOpArg.append(",");
                  finalOpArg.append(tasksFile);
                } else {
                  finalOpArg.append(tasksFile);
                  noneArgsOnly = false;
                }
          

          The tests appear to pass in TestLinuxContainerExecutorWithMocks,
          but it not clear why. One example in TestLinuxContainerExecutorWithMocks that should have caused a test failure :

          StringUtils.join(",", dirsHandler.getLocalDirs()),
          StringUtils.join(",", dirsHandler.getLogDirs()), "cgroups=none"),
          

          It appears to me that this construction is done in enough places that it would make sense to create a static constant for use as a separator when constructing an argument for the container-executor binary. A good candidate location to add such a constant would be the PrivilegedOperation class. You could, in addition, also ‘hide’ the join functionality by adding a static function in the PrivilegedOperation class.

          Show
          sidharta-s Sidharta Seethana added a comment - Wei Yan , you'll need to change PrivilegedOperationExecutor as well if (noneArgsOnly == false ) { //We have already appended at least one tasks file. finalOpArg.append( "," ); finalOpArg.append(tasksFile); } else { finalOpArg.append(tasksFile); noneArgsOnly = false ; } The tests appear to pass in TestLinuxContainerExecutorWithMocks, but it not clear why. One example in TestLinuxContainerExecutorWithMocks that should have caused a test failure : StringUtils.join( "," , dirsHandler.getLocalDirs()), StringUtils.join( "," , dirsHandler.getLogDirs()), "cgroups=none" ), It appears to me that this construction is done in enough places that it would make sense to create a static constant for use as a separator when constructing an argument for the container-executor binary. A good candidate location to add such a constant would be the PrivilegedOperation class. You could, in addition, also ‘hide’ the join functionality by adding a static function in the PrivilegedOperation class.
          Hide
          ywskycn Wei Yan added a comment -

          Karthik Kambatla, Sidharta Seethana, thanks for the comments. Looking into it.

          Show
          ywskycn Wei Yan added a comment - Karthik Kambatla , Sidharta Seethana , thanks for the comments. Looking into it.
          Hide
          ywskycn Wei Yan added a comment -

          Updated a new patch addressing Sidharta Seethana comments. I put the '%' constant to LinuxContainerExecutor.

          Karthik Kambatla, could u try again the latest patch? It runs well in my local test.

          Show
          ywskycn Wei Yan added a comment - Updated a new patch addressing Sidharta Seethana comments. I put the '%' constant to LinuxContainerExecutor. Karthik Kambatla , could u try again the latest patch? It runs well in my local test.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Wei Yan , I see the following in the patch :

          +++ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          @@ -30,6 +30,7 @@
           import org.apache.hadoop.util.Shell.ExitCodeException;
           import org.apache.hadoop.yarn.api.ApplicationConstants;
           import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor;
          

          Please note that LinuxContainerExecutor already has a dependency on PrivilegedOperation and indirectly on PrivilegedOperationExecutor ( via resource handlers ). It doesn't make sense to add a dependency on LinuxContainerExecutor here - it makes it a circular dependency of sorts. PrivilegedOperationExecutor is a wrapper around the container-executor binary and the idea is move all invocations of container-executor to use this mechanism. I recommend instead adding this constant in PrivilegedOperation or PrivilegedOperationExecutor as mentioned before.

          Show
          sidharta-s Sidharta Seethana added a comment - Wei Yan , I see the following in the patch : +++ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java @@ -30,6 +30,7 @@ import org.apache.hadoop.util.Shell.ExitCodeException; import org.apache.hadoop.yarn.api.ApplicationConstants; import org.apache.hadoop.yarn.conf.YarnConfiguration; + import org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor; Please note that LinuxContainerExecutor already has a dependency on PrivilegedOperation and indirectly on PrivilegedOperationExecutor ( via resource handlers ). It doesn't make sense to add a dependency on LinuxContainerExecutor here - it makes it a circular dependency of sorts. PrivilegedOperationExecutor is a wrapper around the container-executor binary and the idea is move all invocations of container-executor to use this mechanism. I recommend instead adding this constant in PrivilegedOperation or PrivilegedOperationExecutor as mentioned before.
          Hide
          ywskycn Wei Yan added a comment -

          Sidharta Seethana, thanks for pointing out. Will fix it soon.

          Show
          ywskycn Wei Yan added a comment - Sidharta Seethana , thanks for pointing out. Will fix it soon.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Wei Yan , the latest patch (v6) looks fine to me. Thanks!

          Varun Vasudev , Vinod Kumar Vavilapalli , Karthik Kambatla could one of you take a quick look and commit please?

          Show
          sidharta-s Sidharta Seethana added a comment - Wei Yan , the latest patch (v6) looks fine to me. Thanks! Varun Vasudev , Vinod Kumar Vavilapalli , Karthik Kambatla could one of you take a quick look and commit please?
          Hide
          vvasudev Varun Vasudev added a comment -

          +1 for the latest patch. Tested it on my machine and it handles the comma issue. I'll commit it tomorrow if there are no objections.

          Show
          vvasudev Varun Vasudev added a comment - +1 for the latest patch. Tested it on my machine and it handles the comma issue. I'll commit it tomorrow if there are no objections.
          Hide
          ywskycn Wei Yan added a comment -

          Thanks, Varun Vasudev.

          Show
          ywskycn Wei Yan added a comment - Thanks, Varun Vasudev .
          Hide
          kasha Karthik Kambatla added a comment -

          I tried the latest patch, and still run into the same issue (logs below). Did anyone try the patch with multiple local directories?

          15/07/01 10:51:32 INFO mapreduce.Job: Job job_1435771879097_0003 failed with state FAILED due to: Application application_1435771879097_0003 failed 2 times due to AM Container for appattempt_1435771879097_0003_000002 exited with  exitCode: -1000
          For more detailed output, check application tracking page:http://krhel7-1.vpc.cloudera.com:8088/proxy/application_1435771879097_0003/Then, click on links to logs of each attempt.
          Diagnostics: Application application_1435771879097_0003 initialization failed (exitCode=20) with output: main : command provided 0
          main : user is nobody
          main : requested yarn user is systest
          Failed to create directory /data/yarn/nm%/data1/yarn/nm/usercache/systest - No such file or directory
          
          Failing this attempt. Failing the application.
          
          Show
          kasha Karthik Kambatla added a comment - I tried the latest patch, and still run into the same issue (logs below). Did anyone try the patch with multiple local directories? 15/07/01 10:51:32 INFO mapreduce.Job: Job job_1435771879097_0003 failed with state FAILED due to: Application application_1435771879097_0003 failed 2 times due to AM Container for appattempt_1435771879097_0003_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://krhel7-1.vpc.cloudera.com:8088/proxy/application_1435771879097_0003/Then, click on links to logs of each attempt. Diagnostics: Application application_1435771879097_0003 initialization failed (exitCode=20) with output: main : command provided 0 main : user is nobody main : requested yarn user is systest Failed to create directory /data/yarn/nm%/data1/yarn/nm/usercache/systest - No such file or directory Failing this attempt. Failing the application.
          Hide
          vvasudev Varun Vasudev added a comment -

          I tested it with multiple local dirs as well. Any chance you can attach the yarn-site.xml you used(or send it to me offline)?

          Show
          vvasudev Varun Vasudev added a comment - I tested it with multiple local dirs as well. Any chance you can attach the yarn-site.xml you used(or send it to me offline)?
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Karthik Kambatla , I have run into such issues when I forgot to rebuild container-executor (requires a different maven profile to be used). So, a shot in the dark : did you re-build the container-executor binary?

          Show
          sidharta-s Sidharta Seethana added a comment - Karthik Kambatla , I have run into such issues when I forgot to rebuild container-executor (requires a different maven profile to be used). So, a shot in the dark : did you re-build the container-executor binary?
          Hide
          kasha Karthik Kambatla added a comment -

          I did a mvn clean package -Pdist -Dtar -Dmaven.javadoc.skip=true -DskipTests -Pnative. I can try changing it to -Pdist,native.

          Show
          kasha Karthik Kambatla added a comment - I did a mvn clean package -Pdist -Dtar -Dmaven.javadoc.skip=true -DskipTests -Pnative . I can try changing it to -Pdist,native.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Karthik Kambatla yes, you need the "native" profile to build container-executor.

          Show
          sidharta-s Sidharta Seethana added a comment - Karthik Kambatla yes, you need the "native" profile to build container-executor.
          Hide
          kasha Karthik Kambatla added a comment -

          yes, you need the "native" profile to build container-executor.

          I did add the native profile (at the end), but I wonder if adding it right after -Pdist would be any different.

          In any case, we should probably go ahead and commit this if multiple people have gotten this to work. I haven't had a chance to look more closely, but it is possible the issues I am seeing are because of some missing commits from our branch. I wasn't using trunk or branch-2.

          Show
          kasha Karthik Kambatla added a comment - yes, you need the "native" profile to build container-executor. I did add the native profile (at the end), but I wonder if adding it right after -Pdist would be any different. In any case, we should probably go ahead and commit this if multiple people have gotten this to work. I haven't had a chance to look more closely, but it is possible the issues I am seeing are because of some missing commits from our branch. I wasn't using trunk or branch-2.
          Hide
          kasha Karthik Kambatla added a comment -

          Varun Vasudev - do you want to do the honors?

          Show
          kasha Karthik Kambatla added a comment - Varun Vasudev - do you want to do the honors?
          Hide
          vvasudev Varun Vasudev added a comment -

          Committed to trunk and branch-2. Thanks Wei!

          Show
          vvasudev Varun Vasudev added a comment - Committed to trunk and branch-2. Thanks Wei!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8123 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8123/)
          YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8123 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8123/ ) YARN-2194 . Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #247 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/)
          YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #247 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/ ) YARN-2194 . Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/)
          YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2195 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/ ) YARN-2194 . Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #250 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/250/)
          YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #250 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/250/ ) YARN-2194 . Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c hadoop-yarn-project/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #980 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/980/)
          YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #980 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/980/ ) YARN-2194 . Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2177/)
          YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2177/ ) YARN-2194 . Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          Hide
          avellant Tatiana Avellanet added a comment -

          Greetings,
          This message is going out to notify that I will be out of the office on July 8 2015 on medical appointments. I will have limited access to either v-mail or e-mail.
          Have a great day,
          Tatiana

          Show
          avellant Tatiana Avellanet added a comment - Greetings, This message is going out to notify that I will be out of the office on July 8 2015 on medical appointments. I will have limited access to either v-mail or e-mail. Have a great day, Tatiana
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #238 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/238/)
          YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #238 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/238/ ) YARN-2194 . Fix bug causing CGroups functionality to fail on RHEL7. Contributed by Wei Yan. (vvasudev: rev c40bdb56a79fe1499c2284d493edc84620c0c078) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -
          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Wei Yan / Varun Vasudev , do we need any additional documentation for this? Say at http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html ?
          Hide
          ywskycn Wei Yan added a comment -

          Vinod Kumar Vavilapalli, Thanks for pointing it out. IMO, I don't think we need additional documentation as the patch doesn't bring new configuration or new implementation mechanism. We need a new documentation when we bring the systemd.

          Show
          ywskycn Wei Yan added a comment - Vinod Kumar Vavilapalli , Thanks for pointing it out. IMO, I don't think we need additional documentation as the patch doesn't bring new configuration or new implementation mechanism. We need a new documentation when we bring the systemd.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Wei Yan , Varun Vasudev

          So, it looks like the final version of the patch that was eventually committed didn't actually go through jenkins ( wasn't submitted to jenkins or something else went wrong during submission ). There seems to be a test failing that needs to be fixed ( see below )

          testSquashCGroupOperationsWithValidOperations(org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor)  Time elapsed: 0.178 sec  <<< FAILURE!
          org.junit.ComparisonFailure: expected:<...n/container_01/tasks[,net_cls/hadoop_yarn/container_01/tasks,]blkio/hadoop_yarn/co...> but was:<...n/container_01/tasks[%net_cls/hadoop_yarn/container_01/tasks%]blkio/hadoop_yarn/co...>
          at org.junit.Assert.assertEquals(Assert.java:115)
          at org.junit.Assert.assertEquals(Assert.java:144)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor.testSquashCGroupOperationsWithValidOperations(TestPrivilegedOperationExecutor.java:225)
          

          thanks,
          -Sidharta

          Show
          sidharta-s Sidharta Seethana added a comment - Wei Yan , Varun Vasudev So, it looks like the final version of the patch that was eventually committed didn't actually go through jenkins ( wasn't submitted to jenkins or something else went wrong during submission ). There seems to be a test failing that needs to be fixed ( see below ) testSquashCGroupOperationsWithValidOperations(org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor) Time elapsed: 0.178 sec <<< FAILURE! org.junit.ComparisonFailure: expected:<...n/container_01/tasks[,net_cls/hadoop_yarn/container_01/tasks,]blkio/hadoop_yarn/co...> but was:<...n/container_01/tasks[%net_cls/hadoop_yarn/container_01/tasks%]blkio/hadoop_yarn/co...> at org.junit.Assert.assertEquals(Assert.java:115) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor.testSquashCGroupOperationsWithValidOperations(TestPrivilegedOperationExecutor.java:225) thanks, -Sidharta
          Hide
          sidharta-s Sidharta Seethana added a comment -

          attaching patch with a fix for unit test issue.

          Show
          sidharta-s Sidharta Seethana added a comment - attaching patch with a fix for unit test issue.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          submitted to jenkins. Vinod Kumar Vavilapalli , please take a quick look and commit ?

          Show
          sidharta-s Sidharta Seethana added a comment - submitted to jenkins. Vinod Kumar Vavilapalli , please take a quick look and commit ?
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 6m 29s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 53s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 38s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 6m 15s Tests passed in hadoop-yarn-server-nodemanager.
              24m 42s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744373/YARN-2194-7.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 2e3d83f
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8465/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8465/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8465/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 6m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 53s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 38s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 13s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 6m 15s Tests passed in hadoop-yarn-server-nodemanager.     24m 42s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744373/YARN-2194-7.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 2e3d83f hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8465/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8465/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8465/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          My apologies for missing the failing unit test Sidharta Seethana. I've committed the fix for the failing unit test.

          Show
          vvasudev Varun Vasudev added a comment - My apologies for missing the failing unit test Sidharta Seethana . I've committed the fix for the failing unit test.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8138 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8138/)
          YARN-2194. Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8138 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8138/ ) YARN-2194 . Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Thanks Varun Vasudev - jenkins wasn't triggered so we all missed it,

          Show
          sidharta-s Sidharta Seethana added a comment - Thanks Varun Vasudev - jenkins wasn't triggered so we all missed it,
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #251 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/251/)
          YARN-2194. Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #251 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/251/ ) YARN-2194 . Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #981 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/981/)
          YARN-2194. Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #981 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/981/ ) YARN-2194 . Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #239 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/239/)
          YARN-2194. Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #239 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/239/ ) YARN-2194 . Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2178 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2178/)
          YARN-2194. Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2178 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2178/ ) YARN-2194 . Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #249 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/249/)
          YARN-2194. Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #249 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/249/ ) YARN-2194 . Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2197 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2197/)
          YARN-2194. Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2197 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2197/ ) YARN-2194 . Addendum patch to fix failing unit test in TestPrivilegedOperationExecutor. Contributed by Sidharta Seethana. (vvasudev: rev 63d0365088ff9fca0baaf3c4c3c01f80c72d3281) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for finding and fixing the issue, Sidharta Seethana. It might be better to create a new JIRA for follow-ups, especially if the person working on it is different; we should avoid the who gets the credit issue if we can

          Show
          kasha Karthik Kambatla added a comment - Thanks for finding and fixing the issue, Sidharta Seethana . It might be better to create a new JIRA for follow-ups, especially if the person working on it is different; we should avoid the who gets the credit issue if we can
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Karthik Kambatla , thanks - will keep that in mind going forward. Assigned this JIRA back to Wei since he did the bulk of the work.

          Show
          sidharta-s Sidharta Seethana added a comment - Karthik Kambatla , thanks - will keep that in mind going forward. Assigned this JIRA back to Wei since he did the bulk of the work.
          Hide
          kasha Karthik Kambatla added a comment -

          No problem at all. I really wish JIRA allows assigning stuff to multiple people, but well there are so many things it could do better.

          Show
          kasha Karthik Kambatla added a comment - No problem at all. I really wish JIRA allows assigning stuff to multiple people, but well there are so many things it could do better.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Varun Vasudev / Sidharta Seethana / Karthik Kambatla, we could just open a placeholder JIRA ticket against Sid for the addendum, post the test fix there and close it right away.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Varun Vasudev / Sidharta Seethana / Karthik Kambatla , we could just open a placeholder JIRA ticket against Sid for the addendum, post the test fix there and close it right away.
          Hide
          carlhe carlhe added a comment -

          My hadoop version is 2.6.4 which is also affected by such bug. Obviously, the patch from (v5) is not available for me, because 2.6.4 even not has the directory "hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/..."
          How can I solve this issue? Thank you~

          Show
          carlhe carlhe added a comment - My hadoop version is 2.6.4 which is also affected by such bug. Obviously, the patch from (v5) is not available for me, because 2.6.4 even not has the directory "hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/..." How can I solve this issue? Thank you~

            People

            • Assignee:
              ywskycn Wei Yan
              Reporter:
              ywskycn Wei Yan
            • Votes:
              0 Vote for this issue
              Watchers:
              37 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development