Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9953 Improve RPC server throughput
  3. HADOOP-9955

RPC idle connection closing is extremely inefficient

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.0-alpha, 3.0.0-alpha1
    • 0.23.10, 2.3.0
    • ipc
    • None
    • Reviewed

    Description

      The RPC server listener loops accepting connections, distributing the new connections to socket readers, and then conditionally & periodically performs a scan for idle connections. The idle scan choses a random index range to scan in a synchronized linked list.

      With 20k+ connections, walking the range of indices in the linked list is extremely expensive. During the sweep, other threads (socket responder and readers) that want to close connections are blocked, and no new connections are being accepted.

      Attachments

        1. HADOOP-9955.patch
          9 kB
          Daryn Sharp
        2. HADOOP-9955.patch
          23 kB
          Daryn Sharp
        3. HADOOP-9955.branch-23.patch
          23 kB
          Daryn Sharp

        Activity

          People

            daryn Daryn Sharp
            daryn Daryn Sharp
            Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: