[HBASE-16393] Improve computeHDFSBlocksDistribution

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Assignee: Lijin Bin
• Reporter: Lijin Bin
• Created: 11/Aug/16 00:42
• Updated: 01/Jul/22 20:23
• Resolved: 08/Apr/20 01:30

Description

With our cluster is big, i can see the balancer is slow from time to time. And the balancer will be called on master startup, so we can see the startup is slow also.
The first thing i think whether if we can parallel compute different region's HDFSBlocksDistribution.
The second i think we can improve compute single region's HDFSBlocksDistribution.
When to compute a storefile's HDFSBlocksDistribution first we call FileSystem#getFileStatus(path) and then FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc call for every storefile. Instead we can use FileSystem#listLocatedStatus to get a LocatedFileStatus for the information we need, so reduce the namenode rpc call to one. This can speed the computeHDFSBlocksDistribution, but also send out less rpc call to namenode.

Issue Links

• incorporates: HBASE-16397 optimize FSRegionScanner
• Is contained by: HBASE-15161 Umbrella: Miscellaneous improvements from production usage
• relates to: HBASE-14473 Compute region locality in parallel

Sub-Tasks

• optimize HRegion computeHDFSBlocksDistribution: Closed
• optimize FSRegionScanner: Closed
• Compute region locality in parallel at startup: Closed
• backport HBASE-16570 to 1.3.1: Closed