[HDDS-1557] Datanode exits because Ratis fails to shutdown ratis server

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.3.0
• Component/s: Ozone Datanode
• Labels: MiniOzoneChaosCluster TriagePending
• Target Version/s: 1.0.0
• Assignee: Aravindan Vijayan
• Reporter: Mukul Kumar Singh
• Created: 19/May/19 07:34
• Updated: 29/Jun/20 16:39
• Resolved: 29/Jun/20 16:39

Description

Datanode exits because Ratis fails to shutdown ratis server

2019-05-19 12:07:19,276 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(965)) - 80747533-f47c-43de-85b8-e70db448c63f: inconsistency entries. Reply:99930d0a-72ab-4795-a3ac-f3c
fb61ca1bb<-80747533-f47c-43de-85b8-e70db448c63f#3132:FAIL,INCONSISTENCY,nextIndex:9057,term:33,followerCommit:9057
2019-05-19 12:07:19,276 WARN  impl.RaftServerProxy (RaftServerProxy.java:lambda$close$4(320)) - e143b976-ab35-4555-a800-7f05a2b1b738: Failed to close GRPC server
java.io.InterruptedIOException: e143b976-ab35-4555-a800-7f05a2b1b738: shutdown server with port 64605 failed
        at org.apache.ratis.util.IOUtils.toInterruptedIOException(IOUtils.java:48)
        at org.apache.ratis.grpc.server.GrpcService.closeImpl(GrpcService.java:160)
        at org.apache.ratis.server.impl.RaftServerRpcWithProxy.lambda$close$2(RaftServerRpcWithProxy.java:76)
        at org.apache.ratis.util.LifeCycle.lambda$checkStateAndClose$2(LifeCycle.java:231)
        at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:251)
        at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:229)
        at org.apache.ratis.server.impl.RaftServerRpcWithProxy.close(RaftServerRpcWithProxy.java:76)
        at org.apache.ratis.server.impl.RaftServerProxy.lambda$close$4(RaftServerProxy.java:318)
        at org.apache.ratis.util.LifeCycle.lambda$checkStateAndClose$2(LifeCycle.java:231)
        at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:251)
        at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:229)
        at org.apache.ratis.server.impl.RaftServerProxy.close(RaftServerProxy.java:313)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.stop(XceiverServerRatis.java:432)
        at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.stop(OzoneContainer.java:201)
        at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.close(DatanodeStateMachine.java:270)
        at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.stopDaemon(DatanodeStateMachine.java:394)
        at org.apache.hadoop.ozone.HddsDatanodeService.stop(HddsDatanodeService.java:449)
        at org.apache.hadoop.ozone.HddsDatanodeService.terminateDatanode(HddsDatanodeService.java:429)
        at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:208)
        at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:349)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.awaitTermination(ServerImpl.java:282)
        at org.apache.ratis.grpc.server.GrpcService.closeImpl(GrpcService.java:158)
        ... 19 more