[KAFKA-14178] NoOpRecord incorrectly causes high controller queue time metric

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Component/s: controller, kraft, metrics
• Assignee: David Arthur
• Reporter: David Arthur
• Created: 23/Aug/22 21:19
• Updated: 17/Oct/22 23:26
• Resolved: 24/Aug/22 17:17

Description

When a deferred event is added to the queue in ControllerQuorum, we include the total time it sat in the queue as part of the "EventQueueTimeMs" metric in QuorumControllerMetrics.

With the introduction of NoOpRecords, the p99 value for this metric is equal to the frequency that we schedule the no-op records. E.g., if no-op records are scheduled every 5 seconds, we will see p99 EventQueueTimeMs of 5 seconds.

This makes it difficult (impossible) to see if there is some delay in the event processing on the controller.

Issue Links

• links to: GitHub Pull Request #12551