Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15136

Explore different queuing behaviors while busy

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 1.3.0, 2.0.0
    • IPC/RPC, Scheduler
    • None
    • Hide
      Previously RPC request scheduler in HBase had 2 modes in could operate in:https://translate.google.com/?source=gtx

       - simple FIFO
       - "partial" deadline, where deadline constraints are only imposed on long-running scan requests.

      This patch adds new type of scheduler to HBase, based on the research around controlled delay (CoDel) algorithm [1], used in networking to combat bufferbloat, as well as some analysis on generalizing it to generic request queues [2]. The purpose of that work is to prevent long standing call queues caused by discrepancy between request rate and available throughput, caused by kernel/disk IO/networking stalls.

      New RPC scheduler could be enabled by setting hbase.ipc.server.callqueue.type=codel in configuration. Several additional params allow to configure algorithm behavior -

      hbase.ipc.server.callqueue.codel.target.delay
      hbase.ipc.server.callqueue.codel.interval
      hbase.ipc.server.callqueue.codel.lifo.threshold

      [1] Controlling Queue Delay / A modern AQM is just one piece of the solution to bufferbloat. http://queue.acm.org/detail.cfm?id=2209336
      [2] Fail at Scale / Reliability in the face of rapid change. http://queue.acm.org/detail.cfm?id=2839461
      Show
      Previously RPC request scheduler in HBase had 2 modes in could operate in: https://translate.google.com/?source=gtx  - simple FIFO  - "partial" deadline, where deadline constraints are only imposed on long-running scan requests. This patch adds new type of scheduler to HBase, based on the research around controlled delay (CoDel) algorithm [1], used in networking to combat bufferbloat, as well as some analysis on generalizing it to generic request queues [2]. The purpose of that work is to prevent long standing call queues caused by discrepancy between request rate and available throughput, caused by kernel/disk IO/networking stalls. New RPC scheduler could be enabled by setting hbase.ipc.server.callqueue.type=codel in configuration. Several additional params allow to configure algorithm behavior - hbase.ipc.server.callqueue.codel.target.delay hbase.ipc.server.callqueue.codel.interval hbase.ipc.server.callqueue.codel.lifo.threshold [1] Controlling Queue Delay / A modern AQM is just one piece of the solution to bufferbloat. http://queue.acm.org/detail.cfm?id=2209336 [2] Fail at Scale / Reliability in the face of rapid change. http://queue.acm.org/detail.cfm?id=2839461

    Attachments

      1. deadline_scheduler_v_0_2.patch
        40 kB
        Mikhail Antonov
      2. HBASE-15136-1.2.v1.patch
        31 kB
        Mikhail Antonov
      3. HBASE-15136-v2.patch
        31 kB
        Mikhail Antonov

      Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. HBASE-15360 Fix flaky TestSimpleRpcScheduler

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. HBASE-15703 Deadline scheduler needs to return to the client info about skipped calls, not just drop them

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Activity

          People

            mantonov Mikhail Antonov
            eclark Elliott Neil Clark
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: