[HBASE-21518] TestMasterFailoverWithProcedures is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.0.3, 2.1.2
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.2, 2.0.4
• Assignee: Peter Somogyi
• Reporter: Peter Somogyi
• Created: 27/Nov/18 15:45
• Updated: 01/Dec/18 16:00
• Resolved: 01/Dec/18 08:33

Description

TestMasterFailoverWithProcedures test is failing frequently, times out. I faced this failure on 2.0.3RC0 vote and it also appears on multiple flaky dashboards.

branch-2: https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/job/branch-2/2007/

branch-2.1: https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/job/branch-2.1/2002/

branch-2.0: https://builds.apache.org/view/H-L/view/HBase/job/HBase-Flaky-Tests/job/branch-2.0/1988/  

[INFO] Running org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
[ERROR] Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 780.648 s <<< FAILURE! - in org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
[ERROR] org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures  Time elapsed: 749.024 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 780 seconds
	at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.tearDown(TestMasterFailoverWithProcedures.java:86)

[ERROR] org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures  Time elapsed: 749.051 s  <<< ERROR!
java.lang.Exception: Appears to be stuck in thread RS-EventLoopGroup-3-2