[HBASE-24528] Improve balancer decision observability

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 1.7.0, 2.4.0
• Component/s: Admin, Balancer, Operability, shell, UI
• Hadoop Flags: Reviewed

Release Note

Retrieve latest balancer decisions made by LoadBalancers.

Examples:
  hbase> get_balancer_decisions
Retrieve recent balancer decisions with region plans

  hbase> get_balancer_decisions LIMIT => 10
Retrieve 10 most recent balancer decisions with region plans


Config change:

hbase.master.balancer.decision.buffer.enabled:

      Indicates whether active HMaster has ring buffer running for storing
      balancer decisions in FIFO manner with limited entries. The size of
      the ring buffer is indicated by config:
      hbase.master.balancer.decision.queue.size

• Assignee: Viraj Jasani
• Reporter: Andrew Kyle Purtell
• Created: 09/Jun/20 23:56
• Updated: 26/Oct/21 11:44
• Resolved: 16/Oct/20 06:53

Description

We provide detailed INFO and DEBUG level logging of balancer decision factors, outcome, and reassignment planning, as well as similarly detailed logging of the resulting assignment manager activity. However, an operator may need to perform online and interactive observation, debugging, or performance analysis of current balancer activity. Scraping and correlating the many log lines resulting from a balancer execution is labor intensive and has a lot of latency (order of ~minutes to acquire and index, order of ~minutes to correlate).

The balancer should maintain a rolling window of history, e.g. the last 100 region move plans, or last 1000 region move plans submitted to the assignment manager. This history should include decision factor details and weights and costs. The rsgroups balancer may be able to provide fairly simple decision factors, like for example "this table was reassigned to that regionserver group". The underlying or vanilla stochastic balancer on the other hand, after a walk over random assignment plans, will have considered a number of cost functions with various inputs (locality, load, etc.) and multipliers, including custom cost functions. We can devise an extensible class structure that represents explanations for balancer decisions, and for each region move plan that is actually submitted to the assignment manager, we can keep the explanations of all relevant decision factors alongside the other details of the assignment plan like the region name, and the source and destination regionservers.

This history should be available via API for use by new shell commands and admin UI widgets.

The new shell commands and UI widgets can unpack the representation of balancer decision components into human readable output.

Issue Links

• relates to: HBASE-25973 Balancer should explain progress in a better way in log
• relates to: HBASE-25697 StochasticBalancer improvement for large scale clusters
• relates to: HBASE-25666 Explain why balancer is skipping runs
• relates to: HBASE-25906 UI of master-status to show recent history of balancer desicion
• relates to: HBASE-26345 Design doc for HBase NamedQueue framework
• links to: GitHub Pull Request #2261
• links to: GitHub Pull Request #2411

Sub-Tasks

• Generic NamedQueue framework for recent in-memory history: Resolved
• Cleanup of unused RPC APIs in the presence of generic API getLogEntries: Resolved