[HADOOP-13144] Enhancing IPC client throughput via multiple connections per user

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: ipc
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Íñigo Goiri
• Reporter: Jason Kace
• Created: 13/May/16 22:30
• Updated: 27/Jan/24 06:37
• Resolved: 15/Jul/22 21:20

Description

The generic IPC client (org.apache.hadoop.ipc.Client) utilizes a single connection thread for each ConnectionId. The ConnectionId is unique to the connection's remote address, ticket and protocol. Each ConnectionId is 1:1 mapped to a connection thread by the client via a map cache.

The result is to serialize all IPC read/write activity through a single thread for a each user/ticket + address. If a single user makes repeated calls (1k-100k/sec) to the same destination, the IPC client becomes a bottleneck.

Issue Links

• is depended upon by: HDFS-13274 RBF: Extend RouterRpcClient to use multiple sockets
• is depended upon by: HDFS-10467 Router-based HDFS federation
• relates to: HADOOP-18406 Adds alignment context to call path for creating RPC proxy with multiple connections per user.
• links to: GitHub Pull Request #4542