Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14674

Rpc handler / task monitoring seems to be broken after 0.98

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In 0.96, we have the RPC handlers listed as tasks and show them in the web UI as well:

      Tasks:
      ===========================================================
      Task: RpcServer.handler=0,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s
      
      Task: RpcServer.handler=1,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s
      
      Task: RpcServer.handler=2,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s
      

      After pluggable RPC scheduler, the way the tasks work for the handlers got changed. We no longer list idle RPC handlers in the tasks, but we register them dynamically to TaskMonitor through CallRunner. However, the IPC readers are still registered the old way (meaning that idle readers are listed as tasks, but not idle handlers).

      From the javadoc of MonitoredRPCHandlerImpl, it seems that we are NOT optimizing the allocation for the MonitoredTask anymore, but instead allocate one for every RPC call breaking the pattern (See CallRunner.getStatus()).

      /**
       * A MonitoredTask implementation designed for use with RPC Handlers 
       * handling frequent, short duration tasks. String concatenations and object 
       * allocations are avoided in methods that will be hit by every RPC call.
       */
      @InterfaceAudience.Private
      public class MonitoredRPCHandlerImpl extends MonitoredTaskImpl
      

      There is also one more side affect that, since the CallRunner is a per-RPC object and created in the RPC listener thread, the created task ends up having a name "listener" although the actual processing happens in a handler thread. This is obviously very confusing during debugging.

        Attachments

        1. HBASE-14674_v1.patch
          8 kB
          Heng Chen
        2. HBASE-14674_v2.patch
          9 kB
          Heng Chen
        3. HBASE-14674.patch
          7 kB
          Heng Chen

          Issue Links

            Activity

              People

              • Assignee:
                chenheng Heng Chen
                Reporter:
                enis Enis Soztutar
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: