[HBASE-17791] Locality should not be affected for non-faulty region servers at startup

• Type: Improvement
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 1.3.0, 1.0.3, 1.2.4, 1.1.8
• Component/s: Balancer, Region Assignment
• Assignee: Unassigned
• Reporter: Lars George
• Created: 15/Mar/17 21:56
• Updated: 12/May/17 19:27

Description

We seem to have an issue with store file locality as soon as a single server is missing or faulty upon restart. HBASE-15251 is addressing a subset of the problem, ensuring that some remaining files or an empty server do not trigger a full recovery, but is missing further, similar cases. Especially, the case where a server fails to report in before the active master is finished waiting for them to do so, or where servers have been decomissioned (but not removed from the regionservers file), and finally the true case of a dead node.

In case a single node is faulty, the user regions are not assigned as saved in the hbase:meta table, but completely randomized in a round-robin fashion. An additional factor is that in this case the regions are not assigned to the best matching node (the one with a copy of the data locally), but to any node, leaving the locality in shambles.

What is also bad, if the hbase.hstore.min.locality.to.skip.major.compact property is left at the default 0.0f, then an older region that had no writes since the last major compaction happened is just skipped (as expected, usually) and locality stays bad as-is. All reads for those aged-out regions will be network reads. But in any event, having to run a major compaction after a restart is not good anyways.

The issue is the code in AssignmentManager.processDeadServersAndRegionsInTransition(), which is handed a list of dead servers. But it immediately sets the failover flag and the code

    failoverCleanupDone();
    if (!failover) {
      // Fresh cluster startup.
      LOG.info("Clean cluster startup. Don't reassign user regions");
      assignAllUserRegions(allRegions);
    } else {
      LOG.info("Failover! Reassign user regions");
    }

is not triggering the assignment of the regions to those servers that are still present and have all their region data local. What should happen is that only the missing regions are reassigned, just like in the case of a server failing while the cluster is running.

Issue Links

• is related to: HBASE-18036 HBase 1.x : Data locality is not maintained after cluster restart or SSH
• relates to: HBASE-14129 If any regionserver gets shutdown uncleanly during full cluster restart, locality looks to be lost