Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Better tooling for replacing failed brokers and reassigning partitions.
      The output of ./bin/kafka-list-topic.sh makes doing this in bulk painful.
      Tool should output json format acceptable to ./bin/kafka-reassign-partitions.sh

      Spec: bin/kafka-gen-reassignment.sh

      Option Description
      ------ -----------
      --topic <topic> REQUIRED: The topic to be reassigned.
      Defaults to all existing topics.
      (default: )
      --partition <partition> REQUIRED: The partition to be reassigned.
      Defaults to all partitions.
      (default: )
      --from <broker-id> REQUIRED: The broker to reassign the partition from
      --to <broker-id> REQUIRED: The broker to reassign the partition to.
      --zookeeper <urls> REQUIRED: The connection string for
      the zookeeper connection in the form
      host:port. Multiple URLS can be
      given to allow fail-over.

      Workflow: Replacing a broker

      ./bin/kafka-gen-reassignment.sh --zookeeper <zks> --from <failed> --to <new> > reassign.json
      ./bin/kafka-reassign-partitions.sh --zookeeper <zks> --path-to-json-file reassign.json

        Activity

        Hide
        Jun Rao added a comment -

        Yes, it's probably useful to have list-topic output json format.

        Show
        Jun Rao added a comment - Yes, it's probably useful to have list-topic output json format.
        Hide
        Scott Clasen added a comment -

        Yes, agree that is possible, but if you want to do any automation of moving partitions, rotating servers, etc, the output format of ./bin/kafka-list-topic.sh makes doing this painful.

        Cold ./bin/kafka-list-topic.sh be modified to (optionally with --json ?) to output a json format that kafka-reassign-partitions.sh can understand? That would make automating things easier and basically cover the above functionality

        Show
        Scott Clasen added a comment - Yes, agree that is possible, but if you want to do any automation of moving partitions, rotating servers, etc, the output format of ./bin/kafka-list-topic.sh makes doing this painful. Cold ./bin/kafka-list-topic.sh be modified to (optionally with --json ?) to output a json format that kafka-reassign-partitions.sh can understand? That would make automating things easier and basically cover the above functionality
        Hide
        Jun Rao added a comment -

        If we just want to use a new broker to replace a failed one, a simpler solution is to just set the broker.id of the new broker to be the same as the failed one. When you restart the new broker, it will automatically replicate all missing data. No need to run the partition reassignment tool.

        Show
        Jun Rao added a comment - If we just want to use a new broker to replace a failed one, a simpler solution is to just set the broker.id of the new broker to be the same as the failed one. When you restart the new broker, it will automatically replicate all missing data. No need to run the partition reassignment tool.

          People

          • Assignee:
            Unassigned
            Reporter:
            Scott Clasen
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development