[HBASE-19828] Flakey TestRegionsOnMasterOptions.testRegionsOnAllServers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Release Note: Disables TestRegionsOnMasterOptions because Regions on Master does not work reliably; see HBASE-19831.
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 19/Jan/18 19:20
• Updated: 01/Aug/18 06:20
• Resolved: 20/Jan/18 18:53

Description

This test is failing 50% of the time now. We seem to have made it fail more w/ our recent changes. The failure here is a good one, again, a real issue. We can get stuck trying to update meta with a Region state if the cluster is going down and hbase:meta has gone down before our client; we get locked-up retrying the put to hbase:meta for so long, the test times out:

 

Thread 2080 (ProcExecWrkr-7):
State: TIMED_WAITING
Blocked count: 0
Waited count: 5960
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.waitUntilDone(AsyncRequestFutureImpl.java:1228)
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.waitUntilDone(AsyncRequestFutureImpl.java:1197)
org.apache.hadoop.hbase.client.HTable.doBatchWithCallback(HTable.java:485)
org.apache.hadoop.hbase.util.MultiHConnection.processBatchCallback(MultiHConnection.java:122)
org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateRegionLocation(RegionStateStore.java:222)
org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateUserRegionLocation(RegionStateStore.java:209)
org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateRegionLocation(RegionStateStore.java:149)
org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1536)
org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:179)
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:85)
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1456)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1225)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1736)

Issue Links

• is related to: HBASE-19527 Make ExecutorService threads daemon=true.
• is related to: HBASE-19831 Regions on the Master Redux
• links to: Review Board (master)