Kafka
  1. Kafka
  2. KAFKA-646

Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:

      Description

      WIth KAFKA-622, we measure ProducerRequestStats and FetchRequestAndResponseStats at the SyncProducer and SimpleConsumer level respectively. We could also aggregate them in the high level Producer and ZookeeperConsumerConnector level to provide an overall sense of request/response rate/size at the client level. Currently, I am not completely clear about the math that might be necessary for such aggregation or if metrics already provides an API for aggregating stats of the same type.

      We should also address the comments by Jun at KAFKA-622, I am copy pasting them here:

      60. What happens if have 2 instances of Consumers with the same clientid in the same jvm? Does one of them fail because it fails to register metrics? Ditto for Producers.
      61. ConsumerTopicStats: What if a topic is named AllTopics? We use to handle this by adding a - in topic specific stats.
      62. ZookeeperConsumerConnector: Do we need to validate groupid?
      63. ClientId: Does the clientid length need to be different from topic length?
      64. AbstractFetcherThread: When building a fetch request, do we need to pass in brokerInfo as part of the client id? BrokerInfo contains the source broker info and the fetch requests are always made to the source broker.

      1. kafka-646-patch-num1-v1.patch
        65 kB
        Swapnil Ghike
      2. kafka-646-patch-num1-v2.patch
        64 kB
        Swapnil Ghike
      3. kafka-646-patch-num1-v3.patch
        64 kB
        Swapnil Ghike
      4. kafka-646-patch-num1-v4.patch
        64 kB
        Swapnil Ghike
      5. kafka-646-patch-num1-v5.patch
        71 kB
        Swapnil Ghike
      6. kafka-646-patch-num1-v6.patch
        71 kB
        Swapnil Ghike
      7. kafka-646-patch-num1-v7.patch
        70 kB
        Swapnil Ghike

        Activity

        Swapnil Ghike created issue -
        Swapnil Ghike made changes -
        Field Original Value New Value
        Labels bugs
        Swapnil Ghike made changes -
        Assignee Swapnil Ghike [ swapnilghike ]
        Swapnil Ghike made changes -
        Attachment kafka-646-patch-num1-v1.patch [ 12559995 ]
        Swapnil Ghike made changes -
        Attachment kafka-646-patch-num1-v2.patch [ 12560022 ]
        Swapnil Ghike made changes -
        Attachment kafka-646-patch-num1-v3.patch [ 12560279 ]
        Swapnil Ghike made changes -
        Attachment kafka-646-patch-num1-v4.patch [ 12560292 ]
        Swapnil Ghike made changes -
        Attachment kafka-646-patch-num1-v5.patch [ 12560434 ]
        Swapnil Ghike made changes -
        Attachment kafka-646-patch-num1-v6.patch [ 12560529 ]
        Swapnil Ghike made changes -
        Attachment kafka-646-patch-num1-v7.patch [ 12560615 ]
        Neha Narkhede made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Neha Narkhede made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Swapnil Ghike
            Reporter:
            Swapnil Ghike
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development