Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14674

Rpc handler / task monitoring seems to be broken after 0.98



    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17, 2.0.0
    • None
    • None
    • Reviewed


      In 0.96, we have the RPC handlers listed as tasks and show them in the web UI as well:

      Task: RpcServer.handler=0,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s
      Task: RpcServer.handler=1,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s
      Task: RpcServer.handler=2,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s

      After pluggable RPC scheduler, the way the tasks work for the handlers got changed. We no longer list idle RPC handlers in the tasks, but we register them dynamically to TaskMonitor through CallRunner. However, the IPC readers are still registered the old way (meaning that idle readers are listed as tasks, but not idle handlers).

      From the javadoc of MonitoredRPCHandlerImpl, it seems that we are NOT optimizing the allocation for the MonitoredTask anymore, but instead allocate one for every RPC call breaking the pattern (See CallRunner.getStatus()).

       * A MonitoredTask implementation designed for use with RPC Handlers 
       * handling frequent, short duration tasks. String concatenations and object 
       * allocations are avoided in methods that will be hit by every RPC call.
      public class MonitoredRPCHandlerImpl extends MonitoredTaskImpl

      There is also one more side affect that, since the CallRunner is a per-RPC object and created in the RPC listener thread, the created task ends up having a name "listener" although the actual processing happens in a handler thread. This is obviously very confusing during debugging.


        1. HBASE-14674_v1.patch
          8 kB
          Heng Chen
        2. HBASE-14674_v2.patch
          9 kB
          Heng Chen
        3. HBASE-14674.patch
          7 kB
          Heng Chen

        Issue Links



              chenheng Heng Chen
              enis Enis Soztutar
              0 Vote for this issue
              9 Start watching this issue