Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24578

[WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.13, 2.2.5
    • Fix Version/s: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.2.6
    • Component/s: wal
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduce a new parameter "hbase.regionserver.wal.sync.batch.count" to control the wal sync batch size which is equals to "hbase.regionserver.handler.count" by default. It should work well if you use default wal provider---one wal per regionserver. But if you use read/write separated handlers, you can set "hbase.regionserver.wal.sync.batch.count" to the number of write handlers. And if you use wal-per-groups or wal-per-region, you can consider lower "hbase.regionserver.wal.sync.batch.count", the default number will be too big and consume more memories as the number of wals grows.
      Show
      Introduce a new parameter "hbase.regionserver.wal.sync.batch.count" to control the wal sync batch size which is equals to "hbase.regionserver.handler.count" by default. It should work well if you use default wal provider---one wal per regionserver. But if you use read/write separated handlers, you can set "hbase.regionserver.wal.sync.batch.count" to the number of write handlers. And if you use wal-per-groups or wal-per-region, you can consider lower "hbase.regionserver.wal.sync.batch.count", the default number will be too big and consume more memories as the number of wals grows.

      Description

      The current value of RingBufferEventHandler's handler is the value of hbase.regionserver.handler.count, which works good in default wal provider — one WAL per regionserver.

      When trying to use WAL group provider, either by group or wal per region, the default value is bad. If rs has 100 regions and wal per region strategy is used, then rs will allocate 100 * SyncFuture[$hbase.regionserver.handler.count] array

      int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT, 200);
          this.ringBufferEventHandler = new RingBufferEventHandler(
              conf.getInt("hbase.regionserver.hlog.syncer.count", 5), maxHandlersCount); 
      ...
      
      ----
      
      RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount) {
            this.syncFutures = new SyncFuture[maxHandlersCount];
            ...
       }
      

        Attachments

          Activity

            People

            • Assignee:
              wenfeiyi666 wenfeiyi666
              Reporter:
              reidchan Reid Chan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: