Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19121 HBCK for AMv2 (A.K.A HBCK2)
  3. HBASE-21191

Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.2.0, 2.1.1, 2.0.3
    • Component/s: amv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Puts master startup into holding pattern if meta is not assigned (previous it would exit). To make progress again, operator needs to inject an assign (Caveats and instruction can be found in HBASE-21035).
      Show
      Puts master startup into holding pattern if meta is not assigned (previous it would exit). To make progress again, operator needs to inject an assign (Caveats and instruction can be found in HBASE-21035 ).

      Description

      If the masterprocwals have been removed – operator error, hdfs dataloss, or because we have gotten ourselves into a pathological state where we have hundreds of masterprocwals too process and it is taking too long so we just want to startover – then master startup will have a dilemma. Master startup needs hbase:meta to be online. If the masterprocwals have been removed, there may be no outstanding assign or a servercrashprocedure with coverage for hbase:meta (I ran into this issue repeatedly in internal testing purging masterprocwals on a large test cluster). Worse, when master startup cannot find an online hbase:meta, it exits after exhausting the RPC retries.

      So, we need a holding-pattern for master startup if hbase:meta is not online if only so an operator can schedule an assign for meta or so they can assign fixup procedures (HBASE-21035 has discussion on why we cannot just auto-schedule an assign of meta).

        Attachments

        1. HBASE-21191.branch-2.0.001.patch
          25 kB
          stack
        2. HBASE-21191.branch-2.1.001.patch
          28 kB
          stack
        3. HBASE-21191.branch-2.1.002.patch
          28 kB
          stack
        4. HBASE-21191.branch-2.1.003.patch
          27 kB
          stack
        5. HBASE-21191.branch-2.1.004.patch
          27 kB
          stack
        6. HBASE-21191.branch-2.1.005.patch
          30 kB
          stack
        7. HBASE-21191.branch-2.1.006.patch
          31 kB
          stack
        8. HBASE-21191.branch-2.1.007.patch
          30 kB
          stack

          Activity

            People

            • Assignee:
              stack stack
              Reporter:
              stack stack
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: