Critical Description
Datanodes are registered fine on a Hadoop + Ozone cluster.
While running jobs against Ozone, Datanode shuts down as below:
2018-10-10 21:50:42,708 INFO storage.RaftLogWorker (RaftLogWorker.java:rollLogSegment(263)) - Rolling segment:7c1a32b5-34ed-4a2a-aa07-ac75d25858b6-RaftLogWorker index to:2 2018-10-10 21:50:42,714 INFO impl.RaftServerImpl (ServerState.java:setRaftConf(319)) - 7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: set configuration 2: [7c1a32b5-34ed-4a2a-aa07-ac75d25858b6:172.27.56.9:9858, ee 20b6291-d898-46de-8cb2-861523aed1a3:172.27.87.64:9858, b7fbd501-27ae-4304-8c42-a612915094c6:172.27.17.133:9858], old=null at 2 2018-10-10 21:50:42,729 WARN impl.LogAppender (LogUtils.java:warn(135)) - 7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: Failed appendEntries to e20b6291-d898-46de-8cb2-861523aed1a3:172.27.87.64:9858: org.apache.. ratis.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-10-10 21:50:43,245 WARN impl.LogAppender (LogUtils.java:warn(135)) - 7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: Failed appendEntries to e20b6291-d898-46de-8cb2-861523aed1a3:172.27.87.64:9858: org.apache.. ratis.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-10-10 21:50:43,310 ERROR impl.RaftServerImpl (RaftServerImpl.java:applyLogToStateMachine(1153)) - 7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: applyTransaction failed for index:1 proto:(t:2, i:1)SMLOGENTRY,, client-894EC0846FDF, cid=0 2018-10-10 21:50:43,313 ERROR impl.StateMachineUpdater (ExitUtils.java:terminate(86)) - Terminating with exit status 2: StateMachineUpdater-7c1a32b5-34ed-4a2a-aa07-ac75d25858b6: the StateMachineUpdater hii ts Throwable java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/management/ObjectName; at org.apache.hadoop.utils.RocksDBStore.<init>(RocksDBStore.java:74) at org.apache.hadoop.utils.MetadataStoreBuilder.build(MetadataStoreBuilder.java:142) at org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.createContainerMetaData(KeyValueContainerUtil.java:78) at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:133) at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:256) at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:179) at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:142) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:223) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:229) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.access$300(ContainerStateMachine.java:115) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine$StateMachineHelper.handleCreateContainer(ContainerStateMachine.java:618) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine$StateMachineHelper.executeContainerCommand(ContainerStateMachine.java:642) at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.applyTransaction(ContainerStateMachine.java:396) at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1150) at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:148) at java.lang.Thread.run(Thread.java:748) 2018-10-10 21:50:43,320 INFO datanode.DataNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at ctr-e138-1518143905142-510793-01-000002.hwx.site/172.27.56.9 ************************************************************/