[HDDS-3719] Datanode may fail to stop

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Ozone Datanode
• Labels: Triaged
• Target Version/s: 1.0.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 04/Jun/20 13:59
• Updated: 05/Jun/20 16:32
• Resolved: 05/Jun/20 16:29

Description

TestOzoneManagerListVolumes timed out after 2 minutes while trying to stop the mini cluster. It seems one of the datanodes was stuck in an infinite loop trying to execute a task on a terminated executor:

2020-06-03 15:28:19,475 [Datanode State Machine Thread - 0] ERROR statemachine.DatanodeStateMachine (DatanodeStateMachine.java:start(221)) - Unable to finish the execution.
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ExecutorCompletionService$QueueingFuture@73c36b6c rejected from org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor@5c021707[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
	at org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState.execute(RunningDatanodeState.java:143)
	at org.apache.hadoop.ozone.container.common.statemachine.StateContext.execute(StateContext.java:411)
	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:208)
	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:375)
	at java.lang.Thread.run(Thread.java:748)

grep -c 'Unable to finish the execution' hadoop-ozone/integration-test/org.apache.hadoop.ozone.om.TestOzoneManagerListVolumes-output.txt

2087250

Test output eventually grew to 2.6GB.

https://github.com/apache/hadoop-ozone/runs/735169623

Issue Links

• links to: GitHub Pull Request #1029