Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 3.3.0
    • controller, kraft, metrics
    • None

    Description

      When a deferred event is added to the queue in ControllerQuorum, we include the total time it sat in the queue as part of the "EventQueueTimeMs" metric in QuorumControllerMetrics.

      With the introduction of NoOpRecords, the p99 value for this metric is equal to the frequency that we schedule the no-op records. E.g., if no-op records are scheduled every 5 seconds, we will see p99 EventQueueTimeMs of 5 seconds.

      This makes it difficult (impossible) to see if there is some delay in the event processing on the controller.

      Attachments

        Activity

          They should contribute to the metrics. The controller needs to make sure to subtract the deferred time from the computed delay.

          You can imagine a large number of delay events that couldn't be process for a while after they triggered.

          jsancio José Armando García Sancio added a comment - They should contribute to the metrics. The controller needs to make sure to subtract the deferred time from the computed delay. You can imagine a large number of delay events that couldn't be process for a while after they triggered.
          cmccabe Colin McCabe made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]
          cmccabe Colin McCabe made changes -
          Status Patch Available [ 10002 ] In Progress [ 3 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #12551 (Web Link)" [ 292706 ] This issue links to "GitHub Pull Request #12551 (Web Link)" [ 292706 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #12551 (Web Link)" [ 292706 ] This issue links to "GitHub Pull Request #12551 (Web Link)" [ 292706 ]
          jagsancio Jose Armando Garcia Sancio made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          jagsancio Jose Armando Garcia Sancio made changes -
          Assignee David Arthur [ mumrah ]
          githubbot ASF GitHub Bot made changes -
          Field Original Value New Value
          Remote Link This issue links to "GitHub Pull Request #12551 (Web Link)" [ 292706 ]
          cmccabe Colin McCabe added a comment -

          it seems like in general, deferred events should not contribute to this metric. NoOpEvent is one of them but there are some others.

          cmccabe Colin McCabe added a comment - it seems like in general, deferred events should not contribute to this metric. NoOpEvent is one of them but there are some others.
          mumrah David Arthur created issue -

          People

            mumrah David Arthur
            mumrah David Arthur
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment