Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17749

Remove lock contention in SelectorPool of SocketIOWithTimeout

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      SelectorPool in hadoop-common/src/main/java/org/apache/hadoop/net/SocketIOWithTimeout.java is a point of lock contention.

      For example: 

      $ grep 'waiting to lock <0x00007f7d94006d90>' 63692.jstack | uniq -c
       1005 - waiting to lock <0x00007f7d94006d90> (a org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
      

      and the thread stack is as follows:

      "IPC Client (324579982) connection to /100.10.6.10:60020 from user_00" #14139 daemon prio=5 os_prio=0 tid=0x00007f7374039000 nid=0x85cc waiting for monitor entry [0x00007f6f45939000]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:390)
       - waiting to lock <0x00007f7d94006d90> (a org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
       at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
       at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
       at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
       at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
       at java.io.FilterInputStream.read(FilterInputStream.java:133)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
       - locked <0x00007fa818caf258> (a java.io.BufferedInputStream)
       at java.io.DataInputStream.readInt(DataInputStream.java:387)
       at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.readResponse(RpcClientImpl.java:967)
       at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)
      
      

      We should remove the lock contention.

        Attachments

          Activity

            People

            • Assignee:
              liangxs Xuesen Liang
              Reporter:
              liangxs Xuesen Liang

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h 50m
                4h 50m

                  Issue deployment