[HBASE-16169] Make RegionSizeCalculator scalable

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: mapreduce, scaling
• Hadoop Flags: Reviewed

Release Note

Added couple of API's to Admin.java:

Returns region load map of all regions hosted on a region server
Map<byte[], RegionLoad> getRegionLoad(ServerName sn) throws IOException;

Returns region load map of all regions of a table hosted on a region server
Map<byte[], RegionLoad> getRegionLoad(ServerName sn, TableName tableName) throws IOException

Added an API to region server:

public GetRegionLoadResponse getRegionLoad(RpcController controller,
    GetRegionLoadRequest request) throws ServiceException;

Primary intention is to use this API for RegionSizeCalculator and not rely on Master for ClusterStatus. On large clusters, ClusterStatus() can take a long time. IfMaster is down/busy, then some of the jobs timeout/fail. Other possible uses:
1. If there is a lighter version of GetClusterStatus API (i.e without the ServerLoad for each RS), then custom maintenance tools can be better. In current world ClusterStatus is heavy. With the new APIs, each API's payload is smaller and distributed. So custom tools can call getRegionLoad() when needed, it will be more accurate. This helps with large clusters. For tools that don't need RegionLoad, the lighter version of API is fine enough.
2. Another use case is a tool like RSTop - since we can see selective metrics at RegionLevel (possibly even deltas between each RPC to the server).

• Assignee: Thiruvel Thirumoolan
• Reporter: Thiruvel Thirumoolan
• Created: 01/Jul/16 18:17
• Updated: 17/Jun/22 15:55
• Resolved: 17/Nov/16 07:08

Description

RegionSizeCalculator is needed for better split generation of MR jobs. This requires RegionLoad which can be obtained via ClusterStatus, i.e. accessing Master. We don't want master to be in this path.

The proposal is to add an API to the RegionServer that gets RegionLoad of all regions hosted on it or those of a table if specified. RegionSizeCalculator can use the latter.

Issue Links

• is blocked by: HBASE-17082 ForeignExceptionUtil isn’t packaged when building shaded protocol with -Pcompile-protobuf
• links to: Review Board (master)
• links to: Review board link