Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-12745

Nodemanagers fail to start because of wrong recovery.dir property

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.1.1
    • Component/s: stacks
    • Labels:
      None

      Description

      $ yarn nodemanager -checkHealth

      15/08/07 15:45:24 INFO nodemanager.NodeManager: STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting NodeManager
      STARTUP_MSG:   host = os-u14-chwavu-oozie-ha-1-5/172.22.126.134
      STARTUP_MSG:   args = [-checkHealth]
      STARTUP_MSG:   version = 2.7.1.2.3.2.0-2602
      STARTUP_MSG:   classpath = /usr/hdp/2.3.2.0-2602/hadoop/conf:/usr/hdp/2.3.2.0-2602/hadoop/conf:/usr/hdp/2.3.2.0-2602/hadoop/conf:....
      
      STARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r f66cf95e2e9367a74b0ec88b2df33458b6cff2d0; compiled by 'jenkins' on 2015-08-05T21:42Z
      STARTUP_MSG:   java = 1.7.0_79
      ************************************************************/
      15/08/07 15:45:24 INFO nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]
      15/08/07 15:45:26 INFO recovery.NMLeveldbStateStoreService: Using state database at /nodemanager/recovery-state/yarn-nm-state for recovery
      15/08/07 15:45:26 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
      org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
      	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
      	at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
      	at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
      	at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
      	at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
      15/08/07 15:45:26 INFO service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
      org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
      	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
      Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
      	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
      	at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
      	at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
      	at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
      	at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      	... 5 more
      15/08/07 15:45:26 FATAL nodemanager.NodeManager: Error starting NodeManager
      org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
      	at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
      	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
      Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
      	at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
      	at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
      	at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
      	at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
      	at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      	... 5 more
      15/08/07 15:45:26 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
      /************************************************************
      SHUTDOWN_MSG: Shutting down NodeManager at os-u14-chwavu-oozie-ha-1-5/172.22.126.134
      ************************************************************/
      yarn@os-u14-chwavu-oozie-ha-1-5:/grid/0/hadoop/yarn$ /usr/hdp/current/hadoop-yarn-nodemanager2015-08-07 01:51:06,160 INFO  nodemanager.NodeManager (LogAdapter.java:info(45)) - STARTUP_MSG:
      /************************************************************
      STARTUP_MSG: Starting NodeManager
      STARTUP_MSG:   host = os-u14-chwavu-oozie-ha-1-5/172.22.126.134
      STARTUP_MSG:   args = []
      STARTUP_MSG:   version = 2.7.1.2.3.2.0-2602
      STARTUP_MSG:   classpath = /usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/2.3.2.0-2602/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/hadoop-lzo-0.6.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-common-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpmime-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/xz-1.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-yarn-plugin-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-audit-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-cred-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-hdfs-plugin-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/zookeeper-3.4.6.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/hadoop-common-2.7.1.2.3.2.0-2602.jar:/usr/hd...skipping...
      /sbin/yarn-daemon.sh --config /tmp/hadoopConf start nodemanager
      starting nodemanager, logging to /grid/0/log/hadoop/yarn/yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.out
      yarn@os-u14-chwavu-oozie-ha-1-5:/grid/0/hadoop/yarn$ ll /grid/0/log/hadoop/yarn/yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.
      yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.log
      yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.out
      2015-08-07 01:51:06,160 INFO  nodemanager.NodeManager (LogAdapter.java:info(45)) - STARTUP_MSG:
      
      

        Attachments

        1. AMBARI-12745.patch
          2 kB
          Dmytro Sen

          Issue Links

            Activity

              People

              • Assignee:
                dsen Dmytro Sen
                Reporter:
                dsen Dmytro Sen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: