Kafka
  1. Kafka
  2. KAFKA-313

Add JSON/CSV output and looping options to ConsumerGroupCommand

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.8.3
    • Component/s: None
    • Labels:

      Description

      Adds:

      • '--loop N' - causes the program to loop forever, sleeping for up to N seconds between loops (loop time minus collection time, unless that's less than 0, at which point it will just run again immediately)
      • '--asjson' - display as a JSON string instead of the more human readable output format.

      Neither of the above depend on each other (you can loop in the human readable output, or do a single shot execution with JSON output). Existing behavior/output maintained if neither of the above are used. Diff Attached.

      Impacted files:

      core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

      1. KAFKA-313_2015-02-23_18:11:32.patch
        9 kB
        Ashish K Singh
      2. KAFKA-313.1.patch
        13 kB
        Ashish K Singh
      3. KAFKA-313.patch
        12 kB
        Ashish K Singh
      4. KAFKA-313-2012032200.diff
        6 kB
        Dave DeMaagd

        Activity

        Dave DeMaagd created issue -
        Dave DeMaagd made changes -
        Field Original Value New Value
        Attachment KAFKA-313-2012032200.diff [ 12519554 ]
        Hide
        Joel Koshy added a comment -

        This tool also needs to be updated for 0.8 for the broker failure and other system tests.

        Show
        Joel Koshy added a comment - This tool also needs to be updated for 0.8 for the broker failure and other system tests.
        Joel Koshy made changes -
        Fix Version/s 0.8 [ 12317244 ]
        Hide
        Jay Kreps added a comment -

        Is JSON better than CSV for this kind of thing? CSV is sometimes more shell friendly and that is what we are using for the perf tests...

        If we like JSON then I am +1

        Show
        Jay Kreps added a comment - Is JSON better than CSV for this kind of thing? CSV is sometimes more shell friendly and that is what we are using for the perf tests... If we like JSON then I am +1
        Hide
        Joel Koshy added a comment -

        I worked with Dave on another version of this patch but will need to revisit. We can add CSV as well.

        Show
        Joel Koshy added a comment - I worked with Dave on another version of this patch but will need to revisit. We can add CSV as well.
        Joe Stein made changes -
        Fix Version/s 0.8.1 [ 12322960 ]
        Fix Version/s 0.8 [ 12317244 ]
        Neha Narkhede made changes -
        Fix Version/s 0.8.2 [ 12326167 ]
        Fix Version/s 0.8.1 [ 12322960 ]
        Hide
        Guozhang Wang added a comment -

        Joel Koshy Do we still need this feature today?

        Show
        Guozhang Wang added a comment - Joel Koshy Do we still need this feature today?
        Neha Narkhede made changes -
        Labels patch newbie patch
        Joe Stein made changes -
        Fix Version/s 0.8.3 [ 12328745 ]
        Fix Version/s 0.8.2 [ 12326167 ]
        Hide
        Ashish K Singh added a comment -

        Joel Koshy I am planning to take a stab at this. If it is OK, then kindly assign this JIRA to me.

        Show
        Ashish K Singh added a comment - Joel Koshy I am planning to take a stab at this. If it is OK, then kindly assign this JIRA to me.
        Hide
        Joel Koshy added a comment -

        I'm not sure if there is a strong need for this. In Linux you can use watch to repeat the command:

        watch -n 10 ./bin/kafka-consumer-offset-checker.sh --zookeeper <zk> --topic <topic> --group <groupid> 2> /dev/null

        Having an in-built loop does save the expense of spinning up a whole VM so it does not hurt to have it I guess.

        Show
        Joel Koshy added a comment - I'm not sure if there is a strong need for this. In Linux you can use watch to repeat the command: watch -n 10 ./bin/kafka-consumer-offset-checker.sh --zookeeper <zk> --topic <topic> --group <groupid> 2> /dev/null Having an in-built loop does save the expense of spinning up a whole VM so it does not hurt to have it I guess.
        Joel Koshy made changes -
        Assignee Ashish Kumar Singh [ singhashish ]
        Joel Koshy made changes -
        Reviewer Joel Koshy [ jjkoshy ]
        Ashish K Singh made changes -
        Summary Add JSON output and looping options to ConsumerOffsetChecker Add JSON/CSV output and looping options to ConsumerOffsetChecker
        Hide
        Ashish K Singh added a comment - - edited

        RB: https://reviews.apache.org/r/28096/
        Adds --loop N and --output.format options. Supports csv and json output formats.

        Show
        Ashish K Singh added a comment - - edited RB: https://reviews.apache.org/r/28096/ Adds --loop N and --output.format options. Supports csv and json output formats.
        Ashish K Singh made changes -
        Attachment KAFKA-313.patch [ 12681753 ]
        Hide
        Ashish K Singh added a comment -

        Joel Koshy while working on this I realized that user needs to know consumer groups available at a given time. I could not find an easy way to do this. A tool to get this information will be handy, created KAFKA-1773.

        Show
        Ashish K Singh added a comment - Joel Koshy while working on this I realized that user needs to know consumer groups available at a given time. I could not find an easy way to do this. A tool to get this information will be handy, created KAFKA-1773 .
        Ashish K Singh made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ashish K Singh made changes -
        Attachment KAFKA-313.1.patch [ 12683259 ]
        Hide
        Ashish K Singh added a comment -

        Joel Koshy can you take a look at the patch?

        Show
        Ashish K Singh added a comment - Joel Koshy can you take a look at the patch?
        Hide
        Ashish K Singh added a comment -

        Joel Koshy, just a reminder , still waiting for your review.

        Show
        Ashish K Singh added a comment - Joel Koshy , just a reminder , still waiting for your review.
        Hide
        Ashish K Singh added a comment -

        Joel Koshy, still waiting for your review.

        Show
        Ashish K Singh added a comment - Joel Koshy , still waiting for your review.
        Hide
        Neha Narkhede added a comment -

        Ashish K Singh We are moving to a centralized ConsumerCommand in KAFKA-1476 that will replace all other consumer admin tools. Can you review to see if the tool has the options you are looking for? Going forward, I think we may want to improve that and phase out the current tools.

        Show
        Neha Narkhede added a comment - Ashish K Singh We are moving to a centralized ConsumerCommand in KAFKA-1476 that will replace all other consumer admin tools. Can you review to see if the tool has the options you are looking for? Going forward, I think we may want to improve that and phase out the current tools.
        Hide
        Gwen Shapira added a comment -

        ConsumerCommand in KAFKA-1476 does not allow looping and does not support JSON output.

        If KAFKA-1476 will be committed soon, it makes sense to re-implement this in the new ConsumerCommand and deprecate the existing ConsumerOffsetChecker.

        Show
        Gwen Shapira added a comment - ConsumerCommand in KAFKA-1476 does not allow looping and does not support JSON output. If KAFKA-1476 will be committed soon, it makes sense to re-implement this in the new ConsumerCommand and deprecate the existing ConsumerOffsetChecker.
        Hide
        Neha Narkhede added a comment -

        If KAFKA-1476 will be committed soon, it makes sense to re-implement this in the new ConsumerCommand and deprecate the existing ConsumerOffsetChecker.

        +1. I'm helping us commit KAFKA-1476

        Show
        Neha Narkhede added a comment - If KAFKA-1476 will be committed soon, it makes sense to re-implement this in the new ConsumerCommand and deprecate the existing ConsumerOffsetChecker. +1. I'm helping us commit KAFKA-1476
        Hide
        Ashish K Singh added a comment -

        Sounds good to me.

        Show
        Ashish K Singh added a comment - Sounds good to me.
        Ashish K Singh made changes -
        Attachment KAFKA-313_2015-02-23_18:11:32.patch [ 12700311 ]
        Hide
        Ashish K Singh added a comment -

        Updated reviewboard https://reviews.apache.org/r/28096/
        against branch trunk

        Show
        Ashish K Singh added a comment - Updated reviewboard https://reviews.apache.org/r/28096/ against branch trunk
        Ashish K Singh made changes -
        Summary Add JSON/CSV output and looping options to ConsumerOffsetChecker Add JSON/CSV output and looping options to ConsumerGroupCommand
        Ashish K Singh made changes -
        Description Adds:
        * '--loop N' - causes the program to loop forever, sleeping for up to N seconds between loops (loop time minus collection time, unless that's less than 0, at which point it will just run again immediately)
        * '--asjson' - display as a JSON string instead of the more human readable output format.

        Neither of the above depend on each other (you can loop in the human readable output, or do a single shot execution with JSON output). Existing behavior/output maintained if neither of the above are used. Diff Attached.

        Impacted files:

        core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
        Adds:
        * '--loop N' - causes the program to loop forever, sleeping for up to N seconds between loops (loop time minus collection time, unless that's less than 0, at which point it will just run again immediately)
        * '--asjson' - display as a JSON string instead of the more human readable output format.

        Neither of the above depend on each other (you can loop in the human readable output, or do a single shot execution with JSON output). Existing behavior/output maintained if neither of the above are used. Diff Attached.

        Impacted files:

        core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala
        Hide
        Ashish K Singh added a comment -

        Gwen Shapira, Neha Narkhede I have updated the patch and the JIRA. Could you guys take a look.

        Show
        Ashish K Singh added a comment - Gwen Shapira , Neha Narkhede I have updated the patch and the JIRA. Could you guys take a look.

          People

          • Assignee:
            Ashish K Singh
            Reporter:
            Dave DeMaagd
            Reviewer:
            Joel Koshy
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development