[HBASE-19965] Fix flaky TestAsyncRegionAdminApi

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.0.0-beta-2, 2.0.0
• Assignee: Michael Stack
• Reporter: Guanghao Zhang
• Created: 09/Feb/18 06:26
• Updated: 21/Mar/18 22:23
• Resolved: 14/Feb/18 01:47

Description

See https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/

 

java.lang.AssertionError: expected:<2> but was:<3> at org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359)

 

Merge regions not work. The table still have 3 regions after the MergeRegionsProcedure finished.

The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because the MergeRegionsProcedure pid=138 start work first, so the balance need wait for the lock. But after merge regions finished, the MoveRegionProcedure pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new region server. This is not right. The MoveRegionProcedure should skip to assign a region which was marked as offline. Or we should clear the merged regions' procedure when MergeRegionsProcedure finished.

 

Logs:

2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] master.HMaster(1454): balance hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., source=cd4730e3eae2,39077,1518106776411, destination=cd4730e3eae2,40578,1518106776318

2018-02-08 16:24:44,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] procedure2.ProcedureExecutor(868): Stored pid=138, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false

......

2018-02-08 16:24:50,111 INFO [PEWorker-13] procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; MergeTableRegionsProcedure table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false in 5.5710sec

2018-02-08 16:24:50,113 INFO [PEWorker-13] procedure.MasterProcedureScheduler(813): pid=139, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., source=cd4730e3eae2,39077,1518106776411, destination=cd4730e3eae2,40578,1518106776318 testMergeRegions testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4.

 

Issue Links

• links to: Review Board (branch-2)