[HBASE-17707] New More Accurate Table Skew cost function/generator

• Type: New Feature
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.2.0
• Component/s: Balancer

Environment

CentOS Derivative with a derivative of the 3.18.43 kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches.

• Hadoop Flags: Reviewed

Release Note

There are now new table skew cost functions and table skew candidate generators in the stochastic load balancer to more evenly spread tables across the cluster. Table skew cost is computed per table, and the final table skew cost number is a weighted average of the maximum skew cost for a given table with the average skew cost across all tables. To configure how much weight the maximum skew cost for a single table should get, you can change "hbase.master.balancer.stochastic.maxTableSkewWeight" to a float between 0.0 and 1.0, where 0.0 means the max table skew gets 0% of the weight and 1.0 means max table skew gets 100% of the weight. This value is useful if you want to strongly penalize any one table being skewed (even if all others are evenly balanced). We default this value to 0.0 because this works best for most cases in practice.

• Flags: Patch
• Assignee: Kahlil Oppenheimer
• Reporter: Kahlil Oppenheimer
• Created: 28/Feb/17 04:53
• Updated: 01/Mar/18 20:29

Description

This patch includes new version of the TableSkewCostFunction and a new TableSkewCandidateGenerator.

The new TableSkewCostFunction computes table skew by counting the minimal number of region moves required for a given table to perfectly balance the table across the cluster (i.e. as if the regions from that table had been round-robin-ed across the cluster). This number of moves is computer for each table, then normalized to a score between 0-1 by dividing by the number of moves required in the absolute worst case (i.e. the entire table is stored on one server), and stored in an array. The cost function then takes a weighted average of the average and maximum value across all tables. The weights in this average are configurable to allow for certain users to more strongly penalize situations where one table is skewed versus where every table is a little bit skewed. To better spread this value more evenly across the range 0-1, we take the square root of the weighted average to get the final value.

The new TableSkewCandidateGenerator generates region moves/swaps to optimize the above TableSkewCostFunction. It first simply tries to move regions until each server has the right number of regions, then it swaps regions around such that each region swap improves table skew across the cluster.

We tested the cost function and generator in our production clusters with 100s of TBs of data and 100s of tables across dozens of servers and found both to be very performant and accurate.

Issue Links

• is related to: HBASE-17706 TableSkewCostFunction improperly computes max skew