[HBASE-17704] Regions stuck in FAILED_OPEN when HDFS blocks are missing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.1.8
• Component/s: regionserver
• Assignee: Unassigned
• Reporter: Mathias Herberts
• Created: 27/Feb/17 22:19
• Updated: 11/Jun/22 23:24
• Resolved: 11/Jun/22 23:24

Description

We recently experienced the loss of a whole rack (6 DNs + RS) in a 120 node cluster. This lead to the regions which were present on the 6 RS which became unavailable to be reassigned to live RSs. When attempting to open some of the reassigned regions, some RS encountered missing blocks and issued "No live nodes contain current block Block locations" putting the regions in state FAILED_OPEN.

Once the disappeared DNs went back online, the regions were left in FAILED_OPEN, needing a restart of all the affected RSs to solve the problem.