Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9640 RPC Congestion Control with FairCallQueue
  3. HADOOP-10597

RPC Server signals backoff to clients when all request queues are full

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 2.7.4, 3.0.0-alpha1
    • None
    • None
    • Reviewed
    • Hide
      This change introduces a new configuration key used by RPC server to decide whether to send backoff signal to RPC Client when RPC call queue is full. When the feature is enabled, RPC server will no longer block on the processing of RPC requests when RPC call queue is full. It helps to improve quality of service when the service is under heavy load. The configuration key is in the format of "ipc.#port#.backoff.enable" where #port# is the port number that RPC server listens on. For example, if you want to enable the feature for the RPC server that listens on 8020, set ipc.8020.backoff.enable to true.
      Show
      This change introduces a new configuration key used by RPC server to decide whether to send backoff signal to RPC Client when RPC call queue is full. When the feature is enabled, RPC server will no longer block on the processing of RPC requests when RPC call queue is full. It helps to improve quality of service when the service is under heavy load. The configuration key is in the format of "ipc.#port#.backoff.enable" where #port# is the port number that RPC server listens on. For example, if you want to enable the feature for the RPC server that listens on 8020, set ipc.8020.backoff.enable to true.

    Description

      Currently if an application hits NN too hard, RPC requests be in blocking state, assuming OS connection doesn't run out. Alternatively RPC or NN can throw some well defined exception back to the client based on certain policies when it is under heavy load; client will understand such exception and do exponential back off, as another implementation of RetryInvocationHandler.

      Attachments

        1. HADOOP-10597.patch
          19 kB
          Ming Ma
        2. RPCClientBackoffDesignAndEvaluation.pdf
          169 kB
          Ming Ma
        3. HADOOP-10597-2.patch
          34 kB
          Ming Ma
        4. MoreRPCClientBackoffEvaluation.pdf
          55 kB
          Ming Ma
        5. HADOOP-10597-3.patch
          32 kB
          Ming Ma
        6. HADOOP-10597-4.patch
          34 kB
          Ming Ma
        7. HADOOP-10597-5.patch
          11 kB
          Ming Ma
        8. HADOOP-10597-6.patch
          11 kB
          Ming Ma
        9. HADOOP-10597-branch-2.7.patch
          13 kB
          Zhe Zhang

        Activity

          People

            mingma Ming Ma
            mingma Ming Ma
            Votes:
            0 Vote for this issue
            Watchers:
            24 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: