[HDDS-689] Datanode shuts down on restart

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Target Version/s: 0.3.0
• Assignee: Mukul Kumar Singh
• Reporter: Namit Maheshwari
• Created: 18/Oct/18 03:31
• Updated: 25/Oct/18 18:56
• Resolved: 25/Oct/18 18:56

Description

Restart all the 8 DNs in the cluster. 2 of them shuts down as below

2018-10-18 01:10:57,102 ERROR impl.StateMachineUpdater (ExitUtils.java:terminate(86)) - Terminating with exit status 2: StateMachineUpdater-69d15283-4e2e-4c30-a028-f2bad0f83cc1: the StateMachineUpdater hits Throwable
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.CompletableFuture$AsyncSupply@7ee4907b rejected from java.util.concurrent.ThreadPoolExecutor@702371d7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
at java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1830)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.applyTransaction(ContainerStateMachine.java:433)
at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1093)
at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:148)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:57,107 WARN fs.CachingGetSpaceUsed (CachingGetSpaceUsed.java:run(183)) - Thread Interrupted waiting to refresh disk information: sleep interrupted
2018-10-18 01:10:57,108 INFO datanode.DataNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG:

 

Issue Links

• is duplicated by: HDDS-716 Update ozone to latest ratis snapshot build(0.3.0-aa38160-SNAPSHOT)