Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-970

Reduce CPU usage of hadoop ipc package

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: ipc
    • Labels:
      None

      Description

      There are a couple of optimizations that could be done to reduce CPU consumption.

      1. The method Server.cleanupConnections() could be invoked less often.
      2. The method Server.cleanupConnections() uses a List to manage all active connections and uses connectionList.get to iterate. Locating the ith element essentially translates to traversing the list from the beginning to the ith position.
      3. The current DFS heartbeattime is 3 seconds whereas ipc.client.connection.maxidletime is set to 1 second. The proposal is to change the default value of ipc.client.connection.maxidletime to something larger than the heartbeat interval. This also has to suit the heartbeat periodicity of map-reduce software.
      4. Evaluate epoll() added in JDK 1.5.10 (this is a java cmd line option) http://java.sun.com/j2se/1.5.0/ReleaseNotes.html#150_10

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                dhruba dhruba borthakur
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: