Blocker Description
$ yarn nodemanager -checkHealth
15/08/07 15:45:24 INFO nodemanager.NodeManager: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NodeManager STARTUP_MSG: host = os-u14-chwavu-oozie-ha-1-5/172.22.126.134 STARTUP_MSG: args = [-checkHealth] STARTUP_MSG: version = 2.7.1.2.3.2.0-2602 STARTUP_MSG: classpath = /usr/hdp/2.3.2.0-2602/hadoop/conf:/usr/hdp/2.3.2.0-2602/hadoop/conf:/usr/hdp/2.3.2.0-2602/hadoop/conf:.... STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r f66cf95e2e9367a74b0ec88b2df33458b6cff2d0; compiled by 'jenkins' on 2015-08-05T21:42Z STARTUP_MSG: java = 1.7.0_79 ************************************************************/ 15/08/07 15:45:24 INFO nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT] 15/08/07 15:45:26 INFO recovery.NMLeveldbStateStoreService: Using state database at /nodemanager/recovery-state/yarn-nm-state for recovery 15/08/07 15:45:26 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573) 15/08/07 15:45:26 INFO service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573) Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 5 more 15/08/07 15:45:26 FATAL nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573) Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 5 more 15/08/07 15:45:26 INFO nodemanager.NodeManager: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NodeManager at os-u14-chwavu-oozie-ha-1-5/172.22.126.134 ************************************************************/ yarn@os-u14-chwavu-oozie-ha-1-5:/grid/0/hadoop/yarn$ /usr/hdp/current/hadoop-yarn-nodemanager2015-08-07 01:51:06,160 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NodeManager STARTUP_MSG: host = os-u14-chwavu-oozie-ha-1-5/172.22.126.134 STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.1.2.3.2.0-2602 STARTUP_MSG: classpath = /usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/2.3.2.0-2602/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/hadoop-lzo-0.6.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-common-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpmime-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/xz-1.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-yarn-plugin-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-audit-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-cred-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-hdfs-plugin-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/zookeeper-3.4.6.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/hadoop-common-2.7.1.2.3.2.0-2602.jar:/usr/hd...skipping... /sbin/yarn-daemon.sh --config /tmp/hadoopConf start nodemanager starting nodemanager, logging to /grid/0/log/hadoop/yarn/yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.out yarn@os-u14-chwavu-oozie-ha-1-5:/grid/0/hadoop/yarn$ ll /grid/0/log/hadoop/yarn/yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5. yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.log yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.out 2015-08-07 01:51:06,160 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - STARTUP_MSG:
Attachments
Attachments
Issue Links
- links to
