Kafka
  1. Kafka
  2. KAFKA-136

A JMX bean that reports #message/sec in consumer

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: None
    • Labels:
      None

      Description

      We need to add a jmx bean that shows # messages consumed by a consumer. The question is whether we want to break this down by topic. If not, the change is pretty straightforward by adding a counter in ConsumerIterator.

        Activity

        Hide
        Neha Narkhede added a comment -

        Thanks for filing this JIRA Jun. It will be very helpful to have this data available. I do think that a per-topic JMX would be helpful.

        Show
        Neha Narkhede added a comment - Thanks for filing this JIRA Jun. It will be very helpful to have this data available. I do think that a per-topic JMX would be helpful.
        Hide
        Jun Rao added a comment -

        Attach a patch for adding a jmx bean that collects cumulated # of messages per topic in the consumer.

        I was trying to add a similar bean in the producer. The best place to add that seems to be in SyncProducer. However, we are forced to iterate all messages in the produce request. This probably will add non-trivial overhead, especially if the request is compressed. Any suggest on how to do this better is welcomed.

        Show
        Jun Rao added a comment - Attach a patch for adding a jmx bean that collects cumulated # of messages per topic in the consumer. I was trying to add a similar bean in the producer. The best place to add that seems to be in SyncProducer. However, we are forced to iterate all messages in the produce request. This probably will add non-trivial overhead, especially if the request is compressed. Any suggest on how to do this better is welcomed.
        Hide
        Neha Narkhede added a comment -

        Patch looks good. Having this information exposed as a JMX bean is useful. I was wondering if we should expose this metric for the SimpleConsumer, since that API is in use too ?

        Show
        Neha Narkhede added a comment - Patch looks good. Having this information exposed as a JMX bean is useful. I was wondering if we should expose this metric for the SimpleConsumer, since that API is in use too ?
        Hide
        Jun Rao added a comment -

        SimpleConsumer is a bit different. We only know that the user get a fetch response. However, we don't know if the messages in the response have been iterated over. So, for SimpleConsumer, we can only collect request level stats.

        Show
        Jun Rao added a comment - SimpleConsumer is a bit different. We only know that the user get a fetch response. However, we don't know if the messages in the response have been iterated over. So, for SimpleConsumer, we can only collect request level stats.

          People

          • Assignee:
            Jun Rao
            Reporter:
            Jun Rao
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development