[HADOOP-9956] RPC listener inefficiently assigns connections to readers

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha, 3.0.0-alpha1
• Fix Version/s: 0.23.10, 2.3.0
• Component/s: ipc
• Hadoop Flags: Reviewed
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 11/Sep/13 23:35
• Updated: 08/Jun/16 03:28
• Resolved: 13/Nov/13 21:49

Description

The socket listener and readers use a complex synchronization to update the reader's NIO Selector. Updating active selectors is not thread-safe so precautions are required.

However, the current locking choreography results in a serialized distribution of new connections to the parallel socket readers. A slower/busier reader can stall the listener and throttle performance.

The problem manifests as unexpectedly low cpu utilization by the listener and readers (~20-30%) under heavy load. The call queue is shallow when it should be overflowing.

Issue Links

• breaks: HADOOP-10406 TestIPC.testIpcWithReaderQueuing may fail