[HDDS-3338] TestOzoneManagerRocksDBLogging.shutdown times out

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Labels: TriagePending
• Target Version/s: 1.4.0
• Sprint: HDDS BadLands
• Assignee: Unassigned
• Reporter: Siyao Meng
• Created: 03/Apr/20 08:00
• Updated: 09/Oct/23 12:27
• Resolved: 09/Oct/23 12:27

Description

Failed in it-hdds-om:

[INFO] Running org.apache.hadoop.ozone.om.TestKeyPurging
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.933 s - in org.apache.hadoop.ozone.om.TestKeyPurging
[INFO] Running org.apache.hadoop.ozone.om.TestOzoneManagerRocksDBLogging
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 60.44 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerRocksDBLogging
[ERROR] testOMRocksDBLoggingEnabled(org.apache.hadoop.ozone.om.TestOzoneManagerRocksDBLogging)  Time elapsed: 60.023 s  <<< ERROR!
java.lang.Exception: test timed out after 60000 milliseconds
	at java.lang.Object.wait(Native Method)
	at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:334)
	at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
	at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
	at org.apache.hadoop.ozone.MiniOzoneClusterImpl.stopDatanodes(MiniOzoneClusterImpl.java:463)
	at org.apache.hadoop.ozone.MiniOzoneClusterImpl.stop(MiniOzoneClusterImpl.java:404)
	at org.apache.hadoop.ozone.MiniOzoneClusterImpl.shutdown(MiniOzoneClusterImpl.java:391)
	at org.apache.hadoop.ozone.om.TestOzoneManagerRocksDBLogging.shutdown(TestOzoneManagerRocksDBLogging.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   TestOzoneManagerRocksDBLogging.shutdown:63->Object.wait:-2 ?  test timed out a...
[INFO] 
[ERROR] Tests run: 62, Failures: 0, Errors: 1, Skipped: 12

Output is FLOODED with the same message:

org.apache.hadoop.ozone.om.TestOzoneManagerRocksDBLogging-output.txt

2020-04-01 20:52:54,072 [Datanode State Machine Thread - 0] ERROR statemachine.DatanodeStateMachine (DatanodeStateMachine.java:start(232)) - Unable to finish the execution.
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ExecutorCompletionService$QueueingFuture@24ae1cc8 rejected from org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor@24ed6a81[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 35]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
	at org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState.execute(RunningDatanodeState.java:144)
	at org.apache.hadoop.ozone.container.common.statemachine.StateContext.execute(StateContext.java:419)
	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:219)
	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:386)
	at java.lang.Thread.run(Thread.java:748)

Artifact: https://github.com/apache/hadoop-ozone/suites/565485547/artifacts/3737405

Could be related to HDDS-2866 ?