Hadoop Common
  1. Hadoop Common
  2. HADOOP-9640 RPC Congestion Control with FairCallQueue
  3. HADOOP-10282

Create a FairCallQueue: a multi-level call queue which schedules incoming calls and multiplexes outgoing calls

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The FairCallQueue ensures quality of service by altering the order of RPC calls internally.

      It consists of three parts:
      1. a Scheduler (`HistoryRpcScheduler` is provided) which provides a priority number from 0 to N (0 being highest priority)
      2. a Multi-level queue (residing in `FairCallQueue`) which provides a way to keep calls in priority order internally
      3. a Multiplexer (`WeightedRoundRobinMultiplexer` is provided) which provides logic to control which queue to take from

      Currently the Mux and Scheduler are not pluggable, but they probably should be (up for discussion).

      This is how it is used:

      // Production
      1. Call is created and given to the CallQueueManager
      2. CallQueueManager requests a `put(T call)` into the `FairCallQueue` which implements `BlockingQueue`
      3. `FairCallQueue` asks its scheduler for a scheduling decision, which is an integer e.g. 12
      4. `FairCallQueue` inserts Call into the 12th queue: `queues.get(12).put(call)`

      // Consumption
      1. CallQueueManager requests `take()` or `poll()` on FairCallQueue
      2. `FairCallQueue` asks its multiplexer for which queue to draw from, which will also be an integer e.g. 2
      3. `FairCallQueue` draws from this queue if it has an available call (or tries other queues if it is empty)

      Additional information is available in the linked JIRAs regarding the Scheduler and Multiplexer's roles.

      1. HADOOP-10282.patch
        29 kB
        Chris Li
      2. HADOOP-10282.patch
        29 kB
        Chris Li
      3. HADOOP-10282.patch
        25 kB
        Chris Li
      4. HADOOP-10282.patch
        25 kB
        Chris Li

        Issue Links

          Activity

          Vinod Kumar Vavilapalli made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arpit Agarwal made changes -
          Fix Version/s 2.6.0 [ 12327179 ]
          Arpit Agarwal made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Chris Li made changes -
          Attachment HADOOP-10282.patch [ 12663541 ]
          Chris Li made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Li made changes -
          Attachment HADOOP-10282.patch [ 12662970 ]
          Chris Li made changes -
          Attachment HADOOP-10282.patch [ 12651311 ]
          Chris Li made changes -
          Assignee Chris Li [ chrilisf ]
          Chris Li made changes -
          Link This issue depends upon HADOOP-10281 [ HADOOP-10281 ]
          Chris Li made changes -
          Link This issue depends upon HADOOP-10279 [ HADOOP-10279 ]
          Chris Li made changes -
          Description The FairCallQueue ensures quality of service by altering the order of RPC calls internally.

          It consists of three parts:
          1. a Scheduler (`HistoryRpcScheduler` is provided) which provides a priority number from 0 to N (0 being highest priority)
          2. a Multi-level queue (residing in `FairCallQueue`) which provides a way to keep calls in priority order internally
          3. a Multiplexer (`WeightedRoundRobinMultiplexer` is provided) which provides logic to control which queue to take from

          Currently the Mux and Scheduler are not pluggable, but they probably should be (up for discussion).

          This is how it is used:

          // Production
          1. Call is created and given to the CallQueueManager
          2. CallQueueManager requests a `put(T call)` into the `FairCallQueue` which implements `BlockingQueue`
          3. `FairCallQueue` asks its scheduler for a scheduling decision, which is an integer e.g. 12
          4. `FairCallQueue` inserts Call into the 12th queue: `queues.get(12).put(call)`

          // Consumption
          1. CallQueueManager requests `take()` or `poll()` on FairCallQueue
          2. `FairCallQueue` asks its multiplexer for which queue to draw from, which will also be an integer e.g. 2
          3. `FairCallQueue` draws from this queue if it has an available call (or tries other queues if it is empty)

          Additional information is available in the linked JIRAs regarding the Scheduler and Multiplexer's roles.
          Chris Li made changes -
          Field Original Value New Value
          Attachment HADOOP-10282.patch [ 12642936 ]
          Chris Li created issue -

            People

            • Assignee:
              Chris Li
              Reporter:
              Chris Li
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development