Hadoop Common
  1. Hadoop Common
  2. HADOOP-970

Reduce CPU usage of hadoop ipc package

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: ipc
    • Labels:
      None

      Description

      There are a couple of optimizations that could be done to reduce CPU consumption.

      1. The method Server.cleanupConnections() could be invoked less often.
      2. The method Server.cleanupConnections() uses a List to manage all active connections and uses connectionList.get to iterate. Locating the ith element essentially translates to traversing the list from the beginning to the ith position.
      3. The current DFS heartbeattime is 3 seconds whereas ipc.client.connection.maxidletime is set to 1 second. The proposal is to change the default value of ipc.client.connection.maxidletime to something larger than the heartbeat interval. This also has to suit the heartbeat periodicity of map-reduce software.
      4. Evaluate epoll() added in JDK 1.5.10 (this is a java cmd line option) http://java.sun.com/j2se/1.5.0/ReleaseNotes.html#150_10

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Unassigned
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development