Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2816

NM fail to start with NPE during container recovery

    Details

    • Hadoop Flags:
      Reviewed

      Description

      NM fail to start with NPE during container recovery.
      We saw the following crash happen:
      2014-10-30 22:22:37,211 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED; cause: java.lang.NullPointerException
      java.lang.NullPointerException
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverContainer(ContainerManagerImpl.java:289)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:252)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:235)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:250)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:445)
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:492)

      The reason is some DB files used in NMLeveldbStateStoreService are accidentally deleted to save disk space at /tmp/hadoop-yarn/yarn-nm-recovery/yarn-nm-state. This leaves some incomplete container record which don't have CONTAINER_REQUEST_KEY_SUFFIX(startRequest) entry in the DB. When container is recovered at ContainerManagerImpl#recoverContainer,
      The NullPointerException at the following code cause NM shutdown.

          StartContainerRequest req = rcs.getStartRequest();
          ContainerLaunchContext launchContext = req.getContainerLaunchContext();
      

        Attachments

        1. YARN-2816.000.patch
          4 kB
          zhihai xu
        2. leveldb_records.txt
          16 kB
          zhihai xu
        3. YARN-2816.001.patch
          4 kB
          zhihai xu
        4. YARN-2816.002.patch
          4 kB
          zhihai xu

          Issue Links

            Activity

              People

              • Assignee:
                zxu zhihai xu
                Reporter:
                zxu zhihai xu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: