[HBASE-20006] TestRestoreSnapshotFromClientWithRegionReplicas is flakey

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.1.0, 1.5.0
• Component/s: read replicas
• Assignee: Toshihiro Suzuki
• Reporter: Michael Stack
• Created: 16/Feb/18 04:34
• Updated: 01/Aug/18 06:23
• Resolved: 20/Apr/18 16:51

Description

Failing 10% of the time. Interestingly, it is below that causes fail. We go to split but it is already split. We will then fail the split with an internal assert which messes up procedures; at a minimum we should just not split (this is in the prepare stage).

2018-02-15 23:21:42,162 INFO  [PEWorker-12] procedure.MasterProcedureScheduler(571): pid=105, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testOnlineSnapshotAfterSplittingRegions-1518736887838, parent=3f850cea7d71a7ebd019f2f009efca4d, daughterA=06b5e6366efbef155d70e56cfdf58dc9, daughterB=8c175de1b33765a5683ac1e502edb0bd, table=testOnlineSnapshotAfterSplittingRegions-1518736887838, testOnlineSnapshotAfterSplittingRegions-1518736887838,,1518736887882.3f850cea7d71a7ebd019f2f009efca4d.
2018-02-15 23:21:42,162 INFO  [PEWorker-12] assignment.SplitTableRegionProcedure(440): Split of {ENCODED => 3f850cea7d71a7ebd019f2f009efca4d, NAME => 'testOnlineSnapshotAfterSplittingRegions-1518736887838,,1518736887882.3f850cea7d71a7ebd019f2f009efca4d.', STARTKEY => '', ENDKEY => '1'} skipped; state is already SPLIT
2018-02-15 23:21:42,163 ERROR [PEWorker-12] procedure2.ProcedureExecutor(1480): CODE-BUG: Uncaught runtime exception: pid=105, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testOnlineSnapshotAfterSplittingRegions-1518736887838, parent=3f850cea7d71a7ebd019f2f009efca4d, daughterA=06b5e6366efbef155d70e56cfdf58dc9, daughterB=8c175de1b33765a5683ac1e502edb0bd
java.lang.AssertionError: split region should have an exception here
  at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:228)
  at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:89)
  at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
  at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)

Issue Links

• links to: Review Board (branch-2)

Sub-Tasks

• Disable TestRestoreSnapshotFromClientWithRegionReplicas; it is flakey. Needs attention.: Resolved