Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14674

Rpc handler / task monitoring seems to be broken after 0.98

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17, 2.0.0
    • None
    • None
    • Reviewed

    Description

      In 0.96, we have the RPC handlers listed as tasks and show them in the web UI as well:

      Tasks:
      ===========================================================
      Task: RpcServer.handler=0,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s
      
      Task: RpcServer.handler=1,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s
      
      Task: RpcServer.handler=2,port=64231
      Status: WAITING:Waiting for a call
      Running for 932s
      

      After pluggable RPC scheduler, the way the tasks work for the handlers got changed. We no longer list idle RPC handlers in the tasks, but we register them dynamically to TaskMonitor through CallRunner. However, the IPC readers are still registered the old way (meaning that idle readers are listed as tasks, but not idle handlers).

      From the javadoc of MonitoredRPCHandlerImpl, it seems that we are NOT optimizing the allocation for the MonitoredTask anymore, but instead allocate one for every RPC call breaking the pattern (See CallRunner.getStatus()).

      /**
       * A MonitoredTask implementation designed for use with RPC Handlers 
       * handling frequent, short duration tasks. String concatenations and object 
       * allocations are avoided in methods that will be hit by every RPC call.
       */
      @InterfaceAudience.Private
      public class MonitoredRPCHandlerImpl extends MonitoredTaskImpl
      

      There is also one more side affect that, since the CallRunner is a per-RPC object and created in the RPC listener thread, the created task ends up having a name "listener" although the actual processing happens in a handler thread. This is obviously very confusing during debugging.

      Attachments

        1. HBASE-14674_v1.patch
          8 kB
          Heng Chen
        2. HBASE-14674_v2.patch
          9 kB
          Heng Chen
        3. HBASE-14674.patch
          7 kB
          Heng Chen

        Issue Links

          Activity

            People

              chenheng Heng Chen
              enis Enis Soztutar
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: