Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1597

New metrics: ResponseQueueSize and BeingSentResponses

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.2.0
    • Component/s: core
    • Labels:

      Description

      This patch adds two metrics:

      ResponseQueueSize

      As of 0.8.1, the sizes of the response queues are reported as different metrics - one per processor thread. This is not very ideal for different reasons:

      • charts have to sum the different metrics
      • the metrics collection system might not support 'wild card queries' like sum:kafka.network.RequestChannel.Processor_*_ResponseQueueSize in which case monitoring now depends on the number of configured network threads
      • monitoring the response by thread is not very valuable. However the global number of responses is useful.
      • proposal*
        So this patch exposes the total number of queued responses as a metric ResponseQueueSize

      implementation
      In RequestChannel, create a Gauge that adds up the size of the response queues.

      BeingSentResponses

      As of 0.8.1, the processor threads will poll responses from the queues and attach them to the SelectionKey as fast as possible. The consequence of that is that the response queues are not a good indicator of the number of "in-flight" responses. The ServerSocketChannel acting as another queue of response to be sent.
      The current metrics don't reflect the size of this "buffer", which is an issue.

      proposal
      This patch adds a gauge that keeps track of the number of responses being handled by the ServerSocketChannel.
      That new metric is named "BeingSentResponses" (who said naming was hard?)

      implementation
      To calculate that metric, the patch adds up the number of SelectionKeys interested in writing, across processor threads.

      Another approach could be to keep all in-flight responses in a data structure (let's say a map) shared by the processor threads. A response will be added to that map when dequeued from the response queue, and removed when the write is complete. The gauge will simply report the size of that map. I decided against that second approach as it is more intrusive and requires some additional bookkeeping to gather information already available through the SelectionKey's

        Attachments

        1. ResponseQueueSize.patch
          0.8 kB
          Alexis Midon
        2. ResponsesBeingSent.patch
          2 kB
          Alexis Midon

          Issue Links

            Activity

              People

              • Assignee:
                alexismidon Alexis Midon
                Reporter:
                alexismidon Alexis Midon
                Reviewer:
                Neha Narkhede
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: