Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14190

Assign system tables ahead of user region assignment

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Region Assignment
    • Labels:
      None

      Description

      Currently the namespace table region is assigned like user regions.

      I spent several hours working with a customer where master couldn't finish initialization.
      Even though master was restarted quite a few times, it went down with the following:

      2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: []
      2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
      java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned
        at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
        at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
        at java.lang.Thread.run(Thread.java:744)
      

      During previous run(s), namespace table was created, hence leaving an entry in hbase:meta.
      The following if block in TableNamespaceManager#start() was skipped:

          if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
            TableName.NAMESPACE_TABLE_NAME)) {
      

      TableNamespaceManager#start() spins, waiting for namespace region to be assigned.
      There was issue in master assigning user regions.
      We tried issuing 'assign' command from hbase shell which didn't work because of the following check in MasterRpcServices#assignRegion():

            master.checkInitialized();
      

      This scenario can be avoided if we assign hbase:namespace table after hbase:meta is assigned but before user table region assignment.

        Attachments

        1. 14190-system-wal-v1.txt
          9 kB
          Ted Yu
        2. 14190-v12.4.txt
          22 kB
          Ted Yu
        3. 14190-v12.txt
          19 kB
          Ted Yu

          Issue Links

            Activity

              People

              • Assignee:
                yuzhihong@gmail.com Ted Yu
                Reporter:
                yuzhihong@gmail.com Ted Yu
              • Votes:
                2 Vote for this issue
                Watchers:
                34 Start watching this issue

                Dates

                • Created:
                  Updated: