[HBASE-26946] TestRetainAssignmentOnRestartSplitWithoutZk infinite loop

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: proc-v2, test
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 12/Apr/22 09:55
• Updated: 17/Apr/22 11:14
• Resolved: 16/Apr/22 05:29

Description

2022-04-05T01:56:10,567 ERROR [Thread-350] server.NIOServerCnxnFactory$1(92): Thread Thread[Thread-350,5,FailOnTimeoutGroup] died
java.lang.IllegalThreadStateException: null
        at java.lang.Thread.start(Thread.java:794) ~[?:?]
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.startWorkers(ProcedureExecutor.java:616) ~[classes/:?]
        at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1666) ~[classes/:?]
        at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart$HMasterForTest.access$101(TestRetainAssignmentOnRestart.java:57) ~[test-classes/:?]
        at org.apache.hadoop.hbase.master.TestRetainAssignmentOnRestart$HMasterForTest.lambda$startProcedureExecutor$0(TestRetainAssignmentOnRestart.java:70) ~[test-classes/:?]
        at java.lang.Thread.run(Thread.java:834) ~[?:?]
2022-04-05T01:56:10,570 INFO  [RS:2;4d77efa4cf73:37349] regionserver.HRegionServer(1179): stopping server 4d77efa4cf73,37349,1649123763933
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [RS:2;4d77efa4cf73:37349] ipc.AbstractRpcClient(505): Stopping rpc client
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 INFO  [RS:2;4d77efa4cf73:37349] regionserver.HRegionServer(1529): Waiting on 1 regions to close
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [RS:2;4d77efa4cf73:37349] regionserver.HRegionServer(1533): Online Regions={3b6c92c681d8cf1316520e3a89546fe0=restartTableThree,,1649123769361.3b6c92c681d8cf1316520e3a89546fe0.}
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running
2022-04-05T01:56:10,570 DEBUG [PEWorker-5] procedure2.AbstractProcedureScheduler(158): the scheduler is not running

After a strange exception then we enter an infinite loop and output a lot of log messages which eat all the spaces...

Issue Links

• is related to: HBASE-26936 project Jenkins nodes need a fail-safe clean up of surefire deferred output
• relates to: HBASE-26947 Implement a special TestAppender to limit the size of test output
• links to: GitHub Pull Request #4344