[HADOOP-2909] Improve IPC idle connection management

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.18.0
• Component/s: ipc
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Removed property ipc.client.maxidletime from the default configuration. The allowed idle time is twice ipc.client.connection.maxidletime.

• Assignee: Hairong Kuang
• Reporter: Hairong Kuang
• Created: 27/Feb/08 18:58
• Updated: 04/Jan/12 04:35
• Resolved: 03/Jun/08 23:09

Description

IPC server determines if a connection is idle or not by checking if the connection does not have any IO activity for a predefined max idle time. An idle connection will be closed even if the connection still has outstanding requests or replies. This causes RPC failures when a server becomes slow or if a request takes a long time to be served. In jira, I'd like to propose the following changes to IPC idle management:
1. Add data structures to the IPC server that keep track of outstanding requests.
2. IPC server does not close a connection that has outstanding requests/replies even when it has no IO activities for a while.
3. The default client-side max idle time should be in several minutes not 1 second.
4. The server-side max idle time should be greater than the client-side max idle time, for example, twice of the client-side max idle time. So server mainly deals with clients that are crashed without closing
its connections.

Issue Links

• is part of: HADOOP-2864 Improve the Scalability and Robustness of IPC
• is related to: HADOOP-7949 Updated maxIdleTime default in the code to match core-default.xml