[HBASE-21191] Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.1, 2.0.3
• Component/s: amv2
• Hadoop Flags: Reviewed

Release Note

Puts master startup into holding pattern if meta is not assigned (previous it would exit). To make progress again, operator needs to inject an assign (Caveats and instruction can be found in HBASE-21035).

• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 12/Sep/18 17:59
• Updated: 04/Mar/19 09:02
• Resolved: 31/Oct/18 19:04

Description

If the masterprocwals have been removed – operator error, hdfs dataloss, or because we have gotten ourselves into a pathological state where we have hundreds of masterprocwals too process and it is taking too long so we just want to startover – then master startup will have a dilemma. Master startup needs hbase:meta to be online. If the masterprocwals have been removed, there may be no outstanding assign or a servercrashprocedure with coverage for hbase:meta (I ran into this issue repeatedly in internal testing purging masterprocwals on a large test cluster). Worse, when master startup cannot find an online hbase:meta, it exits after exhausting the RPC retries.

So, we need a holding-pattern for master startup if hbase:meta is not online if only so an operator can schedule an assign for meta or so they can assign fixup procedures (HBASE-21035 has discussion on why we cannot just auto-schedule an assign of meta).

Issue Links

• links to: Review Board (branch-2.0)
• links to: Review Board (branch-2.1)