Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9953 Improve RPC server throughput
  3. HADOOP-9955

RPC idle connection closing is extremely inefficient

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0-alpha1
    • Fix Version/s: 0.23.10, 2.3.0
    • Component/s: ipc
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The RPC server listener loops accepting connections, distributing the new connections to socket readers, and then conditionally & periodically performs a scan for idle connections. The idle scan choses a random index range to scan in a synchronized linked list.

      With 20k+ connections, walking the range of indices in the linked list is extremely expensive. During the sweep, other threads (socket responder and readers) that want to close connections are blocked, and no new connections are being accepted.

        Attachments

        1. HADOOP-9955.branch-23.patch
          23 kB
          Daryn Sharp
        2. HADOOP-9955.patch
          23 kB
          Daryn Sharp
        3. HADOOP-9955.patch
          9 kB
          Daryn Sharp

          Activity

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: