[HADOOP-17749] Remove lock contention in SelectorPool of SocketIOWithTimeout

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.4.0, 3.2.3, 3.3.2, 3.3.6
• Component/s: common
• Labels: pull-request-available
• Assignee: Xuesen Liang
• Reporter: Xuesen Liang
• Created: 07/Jun/21 04:54
• Updated: 21/Jan/24 15:16
• Resolved: 07/Jul/21 02:07

Description

SelectorPool in hadoop-common/src/main/java/org/apache/hadoop/net/SocketIOWithTimeout.java is a point of lock contention.

For example: 

$ grep 'waiting to lock <0x00007f7d94006d90>' 63692.jstack | uniq -c
 1005 - waiting to lock <0x00007f7d94006d90> (a org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)

and the thread stack is as follows:

"IPC Client (324579982) connection to /100.10.6.10:60020 from user_00" #14139 daemon prio=5 os_prio=0 tid=0x00007f7374039000 nid=0x85cc waiting for monitor entry [0x00007f6f45939000]
 java.lang.Thread.State: BLOCKED (on object monitor)
 at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:390)
 - waiting to lock <0x00007f7d94006d90> (a org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
 at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
 at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
 at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
 at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
 at java.io.FilterInputStream.read(FilterInputStream.java:133)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
 - locked <0x00007fa818caf258> (a java.io.BufferedInputStream)
 at java.io.DataInputStream.readInt(DataInputStream.java:387)
 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.readResponse(RpcClientImpl.java:967)
 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)

We should remove the lock contention.

Issue Links

• links to: GitHub Pull Request #3080