Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11277

RPCServer threads can wedge under high load

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.96.2, 0.98.3
    • Fix Version/s: 0.99.0, 0.96.3, 0.98.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load, RPCServer threads can wedge, fail to make progess, and consume 100% CPU time on a core indefinitely.

      Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE threads are mostly in EPollArrayWrapper.epollWait or FileDispatcherImpl.read0. The number of threads found in FileDispatcherImpl.read0 correspond to the number of runaway threads expected based on looking at 'top' output. These look like:

      Thread 64758: (state = IN_NATIVE)
       - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise)
       - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int) @bci=4, line=39 (Compiled frame)
       - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223 (Compil
      ed frame)
       - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
       - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379 (Compiled frame)
       - org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel, java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
       - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess() @bci=509, line=1488 (Compiled frame)
       - org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey) @bci=23, line=790 (Compiled frame)
       - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97, line=581 (Compiled frame)
       - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1, line=556 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
      

        Attachments

        1. HBASE-11277.patch
          0.8 kB
          Andrew Purtell

          Activity

            People

            • Assignee:
              apurtell Andrew Purtell
              Reporter:
              apurtell Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: