Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11396

Invalid meta entries can lead to unstartable master

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.3
    • Fix Version/s: 0.99.0, 0.98.4
    • Component/s: master
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Recently I accidentally kill -9'd all regionservers in my cluster (don't ask.. not my finest moment).

      This let to some corruption of the meta table, causing the following exception to be output on the HBase Master during startup, and culminating with the Master aborting:

      java.lang.IllegalArgumentException: Wrong length: 13, expected 8
      	at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:600)
      	at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:578)
      	at org.apache.hadoop.hbase.HRegionInfo.getServerName(HRegionInfo.java:1059)
      	at org.apache.hadoop.hbase.HRegionInfo.getHRegionInfoAndServerName(HRegionInfo.java:987)
      	at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2678)
      	at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:465)
      	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:910)
      	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

      No tools that I am aware of were able to clean this up. I added a short patch to catch and log this exception and return null from HRegionInfo.getServerName(). This allowed for startup of the cluster, and hbase hbck to repair the damage.

      Creating ticket to submit patch in case anyone else finds this useful.

        Attachments

        1. HBASE-11396-v1.patch
          1 kB
          Craig Condit

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ccondit Craig Condit
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: