Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15136

Explore different queuing behaviors while busy

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: IPC/RPC, Scheduler
    • Labels:
      None
    • Release Note:
      Hide
      Previously RPC request scheduler in HBase had 2 modes in could operate in:

       - simple FIFO
       - "partial" deadline, where deadline constraints are only imposed on long-running scan requests.

      This patch adds new type of scheduler to HBase, based on the research around controlled delay (CoDel) algorithm [1], used in networking to combat bufferbloat, as well as some analysis on generalizing it to generic request queues [2]. The purpose of that work is to prevent long standing call queues caused by discrepancy between request rate and available throughput, caused by kernel/disk IO/networking stalls.

      New RPC scheduler could be enabled by setting hbase.ipc.server.callqueue.type=codel in configuration. Several additional params allow to configure algorithm behavior -

      hbase.ipc.server.callqueue.codel.target.delay
      hbase.ipc.server.callqueue.codel.interval
      hbase.ipc.server.callqueue.codel.lifo.threshold

      [1] Controlling Queue Delay / A modern AQM is just one piece of the solution to bufferbloat. http://queue.acm.org/detail.cfm?id=2209336
      [2] Fail at Scale / Reliability in the face of rapid change. http://queue.acm.org/detail.cfm?id=2839461
      Show
      Previously RPC request scheduler in HBase had 2 modes in could operate in:  - simple FIFO  - "partial" deadline, where deadline constraints are only imposed on long-running scan requests. This patch adds new type of scheduler to HBase, based on the research around controlled delay (CoDel) algorithm [1], used in networking to combat bufferbloat, as well as some analysis on generalizing it to generic request queues [2]. The purpose of that work is to prevent long standing call queues caused by discrepancy between request rate and available throughput, caused by kernel/disk IO/networking stalls. New RPC scheduler could be enabled by setting hbase.ipc.server.callqueue.type=codel in configuration. Several additional params allow to configure algorithm behavior - hbase.ipc.server.callqueue.codel.target.delay hbase.ipc.server.callqueue.codel.interval hbase.ipc.server.callqueue.codel.lifo.threshold [1] Controlling Queue Delay / A modern AQM is just one piece of the solution to bufferbloat. http://queue.acm.org/detail.cfm?id=2209336 [2] Fail at Scale / Reliability in the face of rapid change. http://queue.acm.org/detail.cfm?id=2839461

      Attachments

      1. deadline_scheduler_v_0_2.patch
        40 kB
        Mikhail Antonov
      2. HBASE-15136-1.2.v1.patch
        31 kB
        Mikhail Antonov
      3. HBASE-15136-v2.patch
        31 kB
        Mikhail Antonov

        Issue Links

          Activity

            People

            • Assignee:
              mantonov Mikhail Antonov
              Reporter:
              eclark Elliott Neil Clark
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: