[YARN-4762] NMs failing on DelegatingLinuxContainerRuntime init with LCE on

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Release Note: Fixed CgroupHandler's creation and usage to avoid NodeManagers crashing when LinuxContainerExecutor is enabled.
• Assignee: Sidharta Seethana
• Reporter: Vinod Kumar Vavilapalli
• Created: 04/Mar/16 02:24
• Updated: 30/Aug/16 01:05
• Resolved: 07/Mar/16 19:12

Description

Seeing this exception and the NMs crash.

2016-03-03 16:47:57,807 DEBUG org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService is started
2016-03-03 16:47:58,027 DEBUG org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: checkLinuxExecutorSetup: [/hadoop/hadoop-yarn-nodemanager/bin/container-executor, --checksetup]
2016-03-03 16:47:58,043 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl: Mount point Based on mtab file: /proc/mounts. Controller mount point not writable for: cpu
2016-03-03 16:47:58,043 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: Unable to get cgroups handle.
2016-03-03 16:47:58,044 DEBUG org.apache.hadoop.service.AbstractService: noteFailure org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
2016-03-03 16:47:58,044 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:240)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:539)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:587)
Caused by: java.io.IOException: Failed to initialize linux container runtime(s)!
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:207)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:238)
        ... 3 more
2016-03-03 16:47:58,047 DEBUG org.apache.hadoop.service.AbstractService: Service: NodeManager entered state STOPPED
2016-03-03 16:47:58,047 DEBUG org.apache.hadoop.service.CompositeService: NodeManager: stopping services, size=0
2016-03-03 16:47:58,047 DEBUG org.apache.hadoop.service.AbstractService: Service: org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService entered state STOPPED
2016-03-03 16:47:58,047 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:240)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:539)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:587)
Caused by: java.io.IOException: Failed to initialize linux container runtime(s)!
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:207)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:238)
        ... 3 more

Issue Links

• is broken by: YARN-4553 Add cgroups support for docker containers