Kafka
  1. Kafka
  2. KAFKA-254

A tool to GET Zookeeper partition-offset and output to files

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.1
    • Component/s: clients
    • Labels:
      None

      Description

      A utility that retrieves the offsets of all topic partitions in ZK for a specified group id and save the data to output files. A shell script also comes with this tool to automate the writing of data to files in a specified time interval.

      This utility expects 3 arguments:
      1. Zk host:port string
      2. group Id
      3. Output file pathname

      1. kafka-254-v1.patch
        8 kB
        John Fung
      2. kafka-254-v2.patch
        5 kB
        John Fung

        Activity

        Hide
        Jun Rao added a comment -

        Thanks for the patch. A few comments:
        1. For consistency, let's use joptsimple for command line processing (take a look at some other tools as examples).
        2. We should use ZkClient, instead of the raw ZK client (see ConsumerOffsetChecker)
        3. Could we extend this to support multiple consumer groups? We can use an option --groups to specify all groups. If the option is not specified, we can get all groups registered in ZK. We will need to include group name in the output file.
        4. Use the zk common paths already defined in ZkUtils.

        Those comments are applicable to kafka-255 too.

        Show
        Jun Rao added a comment - Thanks for the patch. A few comments: 1. For consistency, let's use joptsimple for command line processing (take a look at some other tools as examples). 2. We should use ZkClient, instead of the raw ZK client (see ConsumerOffsetChecker) 3. Could we extend this to support multiple consumer groups? We can use an option --groups to specify all groups. If the option is not specified, we can get all groups registered in ZK. We will need to include group name in the output file. 4. Use the zk common paths already defined in ZkUtils. Those comments are applicable to kafka-255 too.
        Hide
        John Fung added a comment -

        Hi Jun,

        Thanks for your review. Attached kafka-254-v2.patch with the changes suggested in the review.

        Show
        John Fung added a comment - Hi Jun, Thanks for your review. Attached kafka-254-v2.patch with the changes suggested in the review.
        Hide
        Jun Rao added a comment -

        Thanks John. Committed the patch with the following minor changes:
        1. Allow multiple groups specified from command line.
        2. Rename the tool to ExportZkOffsets.
        3. Remove unused imports.

        Show
        Jun Rao added a comment - Thanks John. Committed the patch with the following minor changes: 1. Allow multiple groups specified from command line. 2. Rename the tool to ExportZkOffsets. 3. Remove unused imports.

          People

          • Assignee:
            John Fung
            Reporter:
            John Fung
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development