[HBASE-24578] [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.13, 2.2.5
• Fix Version/s: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.2.6
• Component/s: wal
• Hadoop Flags: Reviewed

Release Note

Introduce a new parameter "hbase.regionserver.wal.sync.batch.count" to control the wal sync batch size which is equals to "hbase.regionserver.handler.count" by default. It should work well if you use default wal provider---one wal per regionserver. But if you use read/write separated handlers, you can set "hbase.regionserver.wal.sync.batch.count" to the number of write handlers. And if you use wal-per-groups or wal-per-region, you can consider lower "hbase.regionserver.wal.sync.batch.count", the default number will be too big and consume more memories as the number of wals grows.

• Assignee: wenfeiyi666
• Reporter: Reid Chan
• Created: 17/Jun/20 04:51
• Updated: 17/Jul/20 11:24
• Resolved: 17/Jul/20 03:10

Description

The current value of RingBufferEventHandler's handler is the value of hbase.regionserver.handler.count, which works good in default wal provider — one WAL per regionserver.

When trying to use WAL group provider, either by group or wal per region, the default value is bad. If rs has 100 regions and wal per region strategy is used, then rs will allocate 100 * SyncFuture[$hbase.regionserver.handler.count] array

int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT, 200);
    this.ringBufferEventHandler = new RingBufferEventHandler(
        conf.getInt("hbase.regionserver.hlog.syncer.count", 5), maxHandlersCount); 
...

----

RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount) {
      this.syncFutures = new SyncFuture[maxHandlersCount];
      ...
 }

Issue Links

• links to: GitHub Pull Request #1923
• links to: GitHub Pull Request #2040
• links to: GitHub Pull Request #2063