[HDDS-824] WriteStateMachineData timesout leading to Datanode crash

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.3.0
• Fix Version/s: 0.3.0
• Assignee: Mukul Kumar Singh
• Reporter: Nilotpal Nandi
• Created: 08/Nov/18 18:24
• Updated: 09/Nov/18 12:49
• Resolved: 09/Nov/18 12:49

Description

datanode stopped due to following error :

datanode.log

2018-10-31 09:12:04,517 INFO org.apache.ratis.server.impl.RaftServerImpl: 9fab9937-fbcd-4196-8014-cb165045724b: set configuration 169: [9fab9937-fbcd-4196-8014-cb165045724b:172.27.15.131:9858, ce0084c2-97cd-4c97-9378-e5175daad18b:172.27.15.139:9858, f0291cb4-7a48-456a-847f-9f91a12aa850:172.27.38.9:9858], old=null at 169
2018-10-31 09:12:22,187 ERROR org.apache.ratis.server.storage.RaftLogWorker: Terminating with exit status 1: 9fab9937-fbcd-4196-8014-cb165045724b-RaftLogWorker failed.
org.apache.ratis.protocol.TimeoutIOException: Timeout: WriteLog:182: (t:10, i:182), STATEMACHINELOGENTRY, client-611073BBFA46, cid=127-writeStateMachineData
 at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:87)
 at org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:310)
 at org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:182)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException
 at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
 at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:79)
 ... 3 more

Issue Links

• is a clone of: HDDS-806 Update Ratis to latest snapshot version in ozone
• is duplicated by: HDDS-783 writeStateMachineData times out (tracking RATIS-382)