[HBASE-18261] [AMv2] Create new RecoverMetaProcedure and use it from ServerCrashProcedure and HMaster.finishActiveMasterInitialization()

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha-1
• Fix Version/s: 2.0.0-alpha-2, 2.0.0
• Component/s: amv2
• Hadoop Flags: Reviewed
• Assignee: Umesh Agashe
• Reporter: Umesh Agashe
• Created: 22/Jun/17 20:55
• Updated: 21/Mar/18 22:10
• Resolved: 31/Jul/17 21:26

Description

When unit test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta() is enabled and run several times, it fails intermittently. Cause is meta recovery is done at two different places:

• ServerCrashProcedure.processMeta()
• HMaster.finishActiveMasterInitialization()
  and its not coordinated.

When HMaster.finishActiveMasterInitialization() gets to submit splitMetaLog() first and while its running call from ServerCrashProcedure.processMeta() fails causing step to be retried again in a loop.

When ServerCrashProcedure.processMeta() submits splitMetaLog after splitMetaLog from HMaster.finishActiveMasterInitialization() is finished, success is returned without doing any work.

But if ServerCrashProcedure.processMeta() submits splitMetaLog request and while its going HMaster.finishActiveMasterInitialization() submits it test fails with exception.

stack and I discussed the possible solution:
Create RecoverMetaProcedure and call it where required. Procedure framework provides mutual exclusion and requires idempotence, which should fix the problem.

Issue Links

• relates to: HBASE-18278 [AMv2] Enable and fix uni test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta
• relates to: HBASE-18366 Fix flaky test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta
• links to: Reviewboard