Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-646

Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.8.0
    • 0.8.0
    • None

    Description

      WIth KAFKA-622, we measure ProducerRequestStats and FetchRequestAndResponseStats at the SyncProducer and SimpleConsumer level respectively. We could also aggregate them in the high level Producer and ZookeeperConsumerConnector level to provide an overall sense of request/response rate/size at the client level. Currently, I am not completely clear about the math that might be necessary for such aggregation or if metrics already provides an API for aggregating stats of the same type.

      We should also address the comments by Jun at KAFKA-622, I am copy pasting them here:

      60. What happens if have 2 instances of Consumers with the same clientid in the same jvm? Does one of them fail because it fails to register metrics? Ditto for Producers.
      61. ConsumerTopicStats: What if a topic is named AllTopics? We use to handle this by adding a - in topic specific stats.
      62. ZookeeperConsumerConnector: Do we need to validate groupid?
      63. ClientId: Does the clientid length need to be different from topic length?
      64. AbstractFetcherThread: When building a fetch request, do we need to pass in brokerInfo as part of the client id? BrokerInfo contains the source broker info and the fetch requests are always made to the source broker.

      Attachments

        1. kafka-646-patch-num1-v1.patch
          65 kB
          Swapnil Ghike
        2. kafka-646-patch-num1-v2.patch
          64 kB
          Swapnil Ghike
        3. kafka-646-patch-num1-v3.patch
          64 kB
          Swapnil Ghike
        4. kafka-646-patch-num1-v4.patch
          64 kB
          Swapnil Ghike
        5. kafka-646-patch-num1-v5.patch
          71 kB
          Swapnil Ghike
        6. kafka-646-patch-num1-v6.patch
          71 kB
          Swapnil Ghike
        7. kafka-646-patch-num1-v7.patch
          70 kB
          Swapnil Ghike

        Activity

          People

            swapnilghike Swapnil Ghike
            swapnilghike Swapnil Ghike
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: