Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6358

Per topic producer/fetch_consumer/fetch_follower metrics

    XMLWordPrintableJSON

    Details

    • Type: Wish
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: metrics
    • Labels:
      None

      Description

      We are using the following JMX beans to monitor Kafka 1.0.0:

      kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce
          Mean
          50thPercentile
          ...
          99thPercentile
      
      kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer
          Count
      
      kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower
          Count
      

      There are more, but this provide an idea of what we are using in order to get produce/fetch operations on a per-broker basis. Nevertheless, in order to identify abusing consumers/clients in our kafka cluster, we would appreciate to have these metrics in a per-topic basis.

      As example of per-topic metrics we have:

      kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=.*
      

      Where we have a per-topic bean with a "Count" attribute that we can query. That way we can know which topics are ingesting more data and which ones less data. We can't do that with the metric explained above.

      Would you consider a change in an upcoming Kafka version as a feature request?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ricbartm Ricardo Bartolome
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: