[HBASE-21325] Force to terminate regionserver when abort hang in somewhere

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.1, 2.0.2
• Fix Version/s: 3.0.0-alpha-1, 1.5.0, 2.2.0

Release Note

Add two new config hbase.regionserver.abort.timeout and hbase.regionserver.abort.timeout.task. If regionserver abort timeout, it will schedule an abort timeout task to run. The default abort task is SystemExitWhenAbortTimeout, which will force to terminate region server when abort timeout. And you can config a special abort timeout task by hbase.regionserver.abort.timeout.task.

• Assignee: Guanghao Zhang
• Reporter: Duo Zhang
• Created: 17/Oct/18 01:44
• Updated: 13/Aug/19 17:26
• Resolved: 29/Oct/18 02:22

Description

When testing sync replication, I found that, if I transit the remote cluster to DA, while the local cluster is still in A, the region server will hang when shutdown. As the fsOk flag only test the local cluster(which is reasonable), we will enter the waitOnAllRegionsToClose, and since the WAL is broken(the remote wal directory is gone) so we will never succeed. And this lead to an infinite wait inside waitOnAllRegionsToClose.

So I think here we should have an upper bound for the wait time in waitOnAllRegionsToClose method.

Issue Links

• relates to: HBASE-21932 Use Runtime.getRuntime().halt to terminate regionserver when abort timeout
• links to: Review Board (master)