[HBASE-12978] Region goes permanently offline (WAS: hbase:meta has a row missing hregioninfo and it causes my long-running job to fail)

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0, 1.1.0
• Hadoop Flags: Reviewed

Release Note

Fixes a bug where an optimization that keeps our hfile indexes files small chanced upon an entry that has special meaning in the meta table making it so we failed to find a region entry in an hbase:meta HFile (though the entry was present). Once we'd happened upon this condition, the region would be unfindable thereafter making it so the region was considered offline. The bug is in branch-1+ hbase only.

• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 05/Feb/15 20:54
• Updated: 06/Apr/18 17:54
• Resolved: 11/Feb/15 01:06

Description

Testing 1.0.0 trying long-running tests.

A row in hbase:meta was missing its HRI entry. It caused the job to fail. Around the time of the first task failure, there are balances of the hbase:meta region and it was on a server that crashed. I tried to look at what happened around time of our writing hbase:meta and I ran into another issue; 20 logs of 256MBs filled with WrongRegionException written over a minute or two. The actual update of hbase:meta was not in the logs, it'd been rotated off.

Sub-Tasks

• Opaque AsyncProcess failure: RetriesExhaustedWithDetailsException but no detail: Closed
• Delete of a table may not clean all rows from hbase:meta: Closed