[HBASE-27389] Add cost function in balancer to consider the cost of building bucket cache before moving regions

• Type: Task
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Component/s: Balancer
• Assignee: Rahul Agarkar
• Reporter: Rahul Agarkar
• Created: 23/Sep/22 16:16
• Updated: 22/Nov/23 13:51

Description

HBase currently uses StochasticLoadBalancer to determine the cost of moving the regions from one RS to another. Each cost functions give a result between 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer iterates through each cost function and comes up with the total cost. Now, the balancer will create multiple balancing plans on random actions and try to compute the cost of each plan as if they are executed, if the cost of the plan is less than the initial cost, the plan is executed.

Implement a new "CacheAwareCostFunction" which takes into account if the region is fully cached and return the highest cost if the plan suggests moving this region.

Issue Links

• links to: GitHub Pull Request #4799
• links to: GitHub Pull Request #5519
• links to: GitHub Pull Request #5533

Sub-Tasks

• Enhance prefetch executor to record region prefetch information along with the list of hfiles prefetched: Resolved
• Enhance region metrics to include prefetch ratio for each region: Resolved
• Implement cache aware load balancer: Resolved
• Rebase HBASE-27389 branch with master and fix conflicts: Resolved
• Reduce frequency of saving backing map in persistence cache: Patch Available
• Add documentation section for the Cache Aware balancer function: Resolved
• Rebase CacheAwareBalance related commits into master branch: Resolved