[HBASE-11396] Invalid meta entries can lead to unstartable master

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.98.3
• Fix Version/s: 0.99.0, 0.98.4
• Component/s: master
• Hadoop Flags: Reviewed
• Assignee: Unassigned
• Reporter: Craig Condit
• Created: 21/Jun/14 18:40
• Updated: 21/Feb/15 23:31
• Resolved: 24/Jun/14 22:50

Description

Recently I accidentally kill -9'd all regionservers in my cluster (don't ask.. not my finest moment).

This let to some corruption of the meta table, causing the following exception to be output on the HBase Master during startup, and culminating with the Master aborting:

java.lang.IllegalArgumentException: Wrong length: 13, expected 8
	at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:600)
	at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:578)
	at org.apache.hadoop.hbase.HRegionInfo.getServerName(HRegionInfo.java:1059)
	at org.apache.hadoop.hbase.HRegionInfo.getHRegionInfoAndServerName(HRegionInfo.java:987)
	at org.apache.hadoop.hbase.master.AssignmentManager.rebuildUserRegions(AssignmentManager.java:2678)
	at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:465)
	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:910)
	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615)
	at java.lang.Thread.run(Thread.java:744)

No tools that I am aware of were able to clean this up. I added a short patch to catch and log this exception and return null from HRegionInfo.getServerName(). This allowed for startup of the cluster, and hbase hbck to repair the damage.

Creating ticket to submit patch in case anyone else finds this useful.