[HBASE-12131] [hbck] undeployRegions should handle gracefully network partitions and other exceptions to avoid the same region deployed multiple times

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Implemented
• Affects Version/s: 0.94.23
• Component/s: hbck
• Assignee: Unassigned
• Reporter: Esteban Gutierrez
• Created: 01/Oct/14 01:10
• Updated: 17/Jun/22 17:31
• Resolved: 11/Jun/22 20:04

Description

If we get an IOE (we currently ignore it) while regions are being undeployed by hbck we should make sure that we don't re-assign that region in the master before we know that RS was marked as dead and optionally let the user to confirm that action or we will end in a split brain situation with clients talking to different RSs serving the same region.

The offending part is here in HBaseFsck.undeployRegions():

 private void undeployRegions(HbckInfo hi) throws IOException, InterruptedException {
    for (OnlineEntry rse : hi.deployedEntries) {
      LOG.debug("Undeploy region "  + rse.hri + " from " + rse.hsa);
      try {
        HBaseFsckRepair.closeRegionSilentlyAndWait(admin, rse.hsa, rse.hri);
        offline(rse.hri.getRegionName());
      } catch (IOException ioe) {
        LOG.warn("Got exception when attempting to offline region "
            + Bytes.toString(rse.hri.getRegionName()), ioe);
      }
    }
  }

Sub-Tasks

• [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires: Closed