Kafka
  1. Kafka
  2. KAFKA-685

ConsumerOffsetChecker does not work with 0.8

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      The ConsumerOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

      Here is a patch to make it work with some cosmetic changes:

      • script-friendly formatting (one line per partition)
      • offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
      • --broker-info optional option to print the broker list at the end (like the previous version)

      Example:

      bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
      Group Topic Pid Offset logSize Lag Owner
      KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
      KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
      KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91

      1. KAFKA-685.patch
        6 kB
        Maxime Brugidou
      2. KAFKA-685-2.patch
        5 kB
        Maxime Brugidou

        Activity

        Maxime Brugidou created issue -
        Maxime Brugidou made changes -
        Field Original Value New Value
        Description The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91

        The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {code}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        {code}
        Maxime Brugidou made changes -
        Description The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {code}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        {code}
        The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {code:bash}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        {code}
        Maxime Brugidou made changes -
        Description The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {code:bash}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        {code}
        The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {code}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        {code}
        Maxime Brugidou made changes -
        Description The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {code}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        {code}
        The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {noformat}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        {noformat}
        Maxime Brugidou made changes -
        Description The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {noformat}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        {noformat}
        The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {{
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        }}
        Maxime Brugidou made changes -
        Description The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {{
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        }}
        The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {{test}}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        }}
        Maxime Brugidou made changes -
        Description The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        {{test}}
        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        }}
        The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        Maxime Brugidou made changes -
        Attachment KAFKA-685.patch [ 12563540 ]
        Maxime Brugidou made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Maxime Brugidou made changes -
        Attachment KAFKA-685-2.patch [ 12563719 ]
        Maxime Brugidou made changes -
        Summary ConsoleOffsetChecker does not work with 0.8 ConsumerOffsetChecker does not work with 0.8
        Maxime Brugidou made changes -
        Description The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        The ConsumerOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker.

        Here is a patch to make it work with some cosmetic changes:

        * script-friendly formatting (one line per partition)
        * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB)
        * --broker-info optional option to print the broker list at the end (like the previous version)

        Example:

        bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181
        Group Topic Pid Offset logSize Lag Owner
        KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0)
        KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0)
        KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91
        Jun Rao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Assignee Maxime Brugidou [ brugidou ]
        Resolution Fixed [ 1 ]
        Jun Rao made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Maxime Brugidou
            Reporter:
            Maxime Brugidou
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development