Details
Description
SelectorPool in hadoop-common/src/main/java/org/apache/hadoop/net/SocketIOWithTimeout.java is a point of lock contention.
For example:
$ grep 'waiting to lock <0x00007f7d94006d90>' 63692.jstack | uniq -c
1005 - waiting to lock <0x00007f7d94006d90> (a org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
and the thread stack is as follows:
"IPC Client (324579982) connection to /100.10.6.10:60020 from user_00" #14139 daemon prio=5 os_prio=0 tid=0x00007f7374039000 nid=0x85cc waiting for monitor entry [0x00007f6f45939000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:390) - waiting to lock <0x00007f7d94006d90> (a org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x00007fa818caf258> (a java.io.BufferedInputStream) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.readResponse(RpcClientImpl.java:967) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)
We should remove the lock contention.
Attachments
Issue Links
- links to