Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4762

NMs failing on DelegatingLinuxContainerRuntime init with LCE on

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed CgroupHandler's creation and usage to avoid NodeManagers crashing when LinuxContainerExecutor is enabled.

      Description

      Seeing this exception and the NMs crash.

      2016-03-03 16:47:57,807 DEBUG org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService is started
      2016-03-03 16:47:58,027 DEBUG org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: checkLinuxExecutorSetup: [/hadoop/hadoop-yarn-nodemanager/bin/container-executor, --checksetup]
      2016-03-03 16:47:58,043 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl: Mount point Based on mtab file: /proc/mounts. Controller mount point not writable for: cpu
      2016-03-03 16:47:58,043 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime: Unable to get cgroups handle.
      2016-03-03 16:47:58,044 DEBUG org.apache.hadoop.service.AbstractService: noteFailure org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
      2016-03-03 16:47:58,044 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:240)
              at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:539)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:587)
      Caused by: java.io.IOException: Failed to initialize linux container runtime(s)!
              at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:207)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:238)
              ... 3 more
      2016-03-03 16:47:58,047 DEBUG org.apache.hadoop.service.AbstractService: Service: NodeManager entered state STOPPED
      2016-03-03 16:47:58,047 DEBUG org.apache.hadoop.service.CompositeService: NodeManager: stopping services, size=0
      2016-03-03 16:47:58,047 DEBUG org.apache.hadoop.service.AbstractService: Service: org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService entered state STOPPED
      2016-03-03 16:47:58,047 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:240)
              at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:539)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:587)
      Caused by: java.io.IOException: Failed to initialize linux container runtime(s)!
              at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:207)
              at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:238)
              ... 3 more
      

        Attachments

        1. YARN-4762.001.patch
          9 kB
          Sidharta Seethana
        2. YARN-4762.002.patch
          12 kB
          Sidharta Seethana

          Issue Links

            Activity

              People

              • Assignee:
                sidharta-s Sidharta Seethana
                Reporter:
                vinodkv Vinod Kumar Vavilapalli
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: