[HADOOP-970] Reduce CPU usage of hadoop ipc package

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: ipc
• Assignee: Unassigned
• Reporter: Dhruba Borthakur
• Created: 02/Feb/07 19:02
• Updated: 16/Jul/11 18:26

Description

There are a couple of optimizations that could be done to reduce CPU consumption.

1. The method Server.cleanupConnections() could be invoked less often.
2. The method Server.cleanupConnections() uses a List to manage all active connections and uses connectionList.get to iterate. Locating the ith element essentially translates to traversing the list from the beginning to the ith position.
3. The current DFS heartbeattime is 3 seconds whereas ipc.client.connection.maxidletime is set to 1 second. The proposal is to change the default value of ipc.client.connection.maxidletime to something larger than the heartbeat interval. This also has to suit the heartbeat periodicity of map-reduce software.
4. Evaluate epoll() added in JDK 1.5.10 (this is a java cmd line option) http://java.sun.com/j2se/1.5.0/ReleaseNotes.html#150_10

Issue Links

• blocks: HADOOP-885 Reduce CPU usage on namenode: gettimeofday