Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22935

TaskMonitor warns MonitoredRPCHandler task may be stuck when it recently started

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      After settingĀ hbase.taskmonitor.rpc.warn.time to 180000, the logs show WARN messages such as these

      2019-08-08 21:50:02,601 WARN  [read for TaskMonitor] monitoring.TaskMonitor - Task may be stuck: RpcServer.FifoWFPBQ.default.handler=4,queue=4,port=60020: status=Servicing call from <ip>:55164: Scan, state=RUNNING, startTime=1563305858103, completionTime=-1, queuetimems=1565301002599, starttimems=1565301002599, clientaddress=<ip>, remoteport=55164, packetlength=370, rpcMethod=Scan
      

      Notice that the first starttimems is far in the past. The second starttimems and the queuetimems are much closer to the log timestamp than 180 seconds. I think this is because the warnTime is initialized to the time that MonitoredTaskImpl is created, but never updated until we write a warn message to the log.

      Attachments

        1. HBASE-22935.master.001.patch
          3 kB
          David Manning

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dmanning David Manning
            dmanning David Manning
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment