Kafka
  1. Kafka
  2. KAFKA-685

ConsumerOffsetChecker does not work with 0.8

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      The ConsumerOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

      Here is a patch to make it work with some cosmetic changes:

      • script-friendly formatting (one line per partition)
      • offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
      • --broker-info optional option to print the broker list at the end (like the previous version)

      Example:

      bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
      Group Topic Pid Offset logSize Lag Owner
      KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
      KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
      KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91

      1. KAFKA-685.patch
        6 kB
        Maxime Brugidou
      2. KAFKA-685-2.patch
        5 kB
        Maxime Brugidou

        Activity

        Hide
        Jun Rao added a comment -

        Thanks for patch v2. Committed to 0.8 with the following tweaks.

        1. Print owner better since it's an Option.

        2. Increased the width of 2 fields in the output.

        Show
        Jun Rao added a comment - Thanks for patch v2. Committed to 0.8 with the following tweaks. 1. Print owner better since it's an Option. 2. Increased the width of 2 fields in the output.
        Hide
        Maxime Brugidou added a comment -

        New patch, updated the regex to ignore JMX port

        Show
        Maxime Brugidou added a comment - New patch, updated the regex to ignore JMX port
        Hide
        Jun Rao added a comment -

        Thanks for the patch. The code looks good. Got the following error on latest 0.8 branch. This is likely due to the recent change in KAFKA-668. So you need to change the regex a bit.

        ERROR Could not parse broker info Some(jrao-ld.linkedin.biz:9092:9999) with regex ^([^:])\d)$ (kafka.tools.ConsumerOffsetChecker$)

        Show
        Jun Rao added a comment - Thanks for the patch. The code looks good. Got the following error on latest 0.8 branch. This is likely due to the recent change in KAFKA-668 . So you need to change the regex a bit. ERROR Could not parse broker info Some(jrao-ld.linkedin.biz:9092:9999) with regex ^( [^:] ) \d )$ (kafka.tools.ConsumerOffsetChecker$)

          People

          • Assignee:
            Maxime Brugidou
            Reporter:
            Maxime Brugidou
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development