Kafka
  1. Kafka
  2. KAFKA-254

A tool to GET Zookeeper partition-offset and output to files

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.1
    • Component/s: clients
    • Labels:
      None

      Description

      A utility that retrieves the offsets of all topic partitions in ZK for a specified group id and save the data to output files. A shell script also comes with this tool to automate the writing of data to files in a specified time interval.

      This utility expects 3 arguments:
      1. Zk host:port string
      2. group Id
      3. Output file pathname

      1. kafka-254-v2.patch
        5 kB
        John Fung
      2. kafka-254-v1.patch
        8 kB
        John Fung

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2d 23h 1 John Fung 27/Jan/12 22:21
        Patch Available Patch Available Resolved Resolved
        24d 2h 6m 1 Jun Rao 21/Feb/12 00:28
        Jun Rao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.7.1 [ 12319140 ]
        Resolution Fixed [ 1 ]
        Hide
        Jun Rao added a comment -

        Thanks John. Committed the patch with the following minor changes:
        1. Allow multiple groups specified from command line.
        2. Rename the tool to ExportZkOffsets.
        3. Remove unused imports.

        Show
        Jun Rao added a comment - Thanks John. Committed the patch with the following minor changes: 1. Allow multiple groups specified from command line. 2. Rename the tool to ExportZkOffsets. 3. Remove unused imports.
        John Fung made changes -
        Attachment kafka-254-v2.patch [ 12514390 ]
        Hide
        John Fung added a comment -

        Hi Jun,

        Thanks for your review. Attached kafka-254-v2.patch with the changes suggested in the review.

        Show
        John Fung added a comment - Hi Jun, Thanks for your review. Attached kafka-254-v2.patch with the changes suggested in the review.
        Hide
        Jun Rao added a comment -

        Thanks for the patch. A few comments:
        1. For consistency, let's use joptsimple for command line processing (take a look at some other tools as examples).
        2. We should use ZkClient, instead of the raw ZK client (see ConsumerOffsetChecker)
        3. Could we extend this to support multiple consumer groups? We can use an option --groups to specify all groups. If the option is not specified, we can get all groups registered in ZK. We will need to include group name in the output file.
        4. Use the zk common paths already defined in ZkUtils.

        Those comments are applicable to kafka-255 too.

        Show
        Jun Rao added a comment - Thanks for the patch. A few comments: 1. For consistency, let's use joptsimple for command line processing (take a look at some other tools as examples). 2. We should use ZkClient, instead of the raw ZK client (see ConsumerOffsetChecker) 3. Could we extend this to support multiple consumer groups? We can use an option --groups to specify all groups. If the option is not specified, we can get all groups registered in ZK. We will need to include group name in the output file. 4. Use the zk common paths already defined in ZkUtils. Those comments are applicable to kafka-255 too.
        John Fung made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        John Fung made changes -
        Attachment kafka-254-v1.patch [ 12512240 ]
        John Fung made changes -
        Field Original Value New Value
        Summary A tool to GET Zookeeper partition-offset and output in JSONObject string A tool to GET Zookeeper partition-offset and output to files
        Description A utility that retrieves the offset of all topic partitions in ZK and save the data as a JSONObject string in the output file.

        This utility expects 2 arguments:
         - 1. Zk host:port string
         - 2. Output file pathname

        ** Please note, this utility requires an external dependency of "JSON in Java" (http://www.json.org/java/index.html)
        A utility that retrieves the offsets of all topic partitions in ZK for a specified group id and save the data to output files. A shell script also comes with this tool to automate the writing of data to files in a specified time interval.

        This utility expects 3 arguments:
         1. Zk host:port string
         2. group Id
         3. Output file pathname
        John Fung created issue -

          People

          • Assignee:
            John Fung
            Reporter:
            John Fung
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development