Kafka
  1. Kafka
  2. KAFKA-822 Reassignment of partitions needs a cleanup
  3. KAFKA-821

specifying a non-existent partition to ./bin/kafka-reassign-partitions.sh breaks all reassignment ops

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: controller, tools
    • Labels:

      Description

      From my ML post...the workaround is to manually delete /admin/reassign_partitions from ZK, that node contained a json with an empty partitions array.

      Have 3 brokers running. Ids 25,26,27

      ./bin/kafka-create-topic.sh --replica 3 --topic first-cluster-topic
      --zookeeper <zk>:2181/kafka

      Seems fine, can send/receive, etc..

      Kill 27, start 28. Try to reassign the single partition topic with the
      following json.

      Contains an error. partition should be 0 not 1.

      {"partitions":
      [

      {"topic": "first-cluster-topic", "partition": 1, "replicas": [25,26,28] }

      ]
      }

      ./bin/kafka-reassign-partitions.sh --zookeeper ... -path-to-json-file
      reassign.json

      2013-03-21 12:14:46,170] INFO zookeeper state changed (SyncConnected)
      (org.I0Itec.zkclient.ZkClient)
      [2013-03-21 12:14:46,310] ERROR Skipping reassignment of partition
      [first-cluster-topic,1] since it doesn't exist
      (kafka.admin.ReassignPartitionsCommand)
      Successfully started reassignment of partitions Map([first-cluster-topic,1]
      -> List(25, 26, 28))
      [2013-03-21 12:14:46,665] INFO Terminate ZkClient event thread.
      (org.I0Itec.zkclient.ZkEventThread)
      [2013-03-21 12:14:46,780] INFO Session: 0x13d8a63a3760007 closed
      (org.apache.zookeeper.ZooKeeper)
      [2013-03-21 12:14:46,780] INFO EventThread shut down
      (org.apache.zookeeper.ClientCnxn)

      Ok, fix the JSON

      {"partitions":
      [

      {"topic": "first-cluster-topic", "partition": 0, "replicas": [25,26,28] }

      ]
      }

      ./bin/kafka-reassign-partitions.sh --zookeeper ... -path-to-json-file
      reassign.json

      [2013-03-21 12:17:34,367] INFO zookeeper state changed (SyncConnected)
      (org.I0Itec.zkclient.ZkClient)
      Partitions reassignment failed due to Partition reassignment currently in
      progress for Map(). Aborting operation
      kafka.common.AdminCommandFailedException: Partition reassignment currently
      in progress for Map(). Aborting operation
      at
      kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:91)
      at
      kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:65)
      at
      kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

      ./bin/kafka-check-reassignment-status.sh --zookeeper ...
      --path-to-json-file reassign.json

      [2013-03-21 12:20:40,607] INFO zookeeper state changed (SyncConnected)
      (org.I0Itec.zkclient.ZkClient)
      Exception in thread "main" java.lang.ClassCastException:
      scala.collection.immutable.Map$Map1 cannot be cast to
      [Lscala.collection.Map;
      at
      kafka.admin.CheckReassignmentStatus$.main(CheckReassignmentStatus.scala:44)
      at kafka.admin.CheckReassignmentStatus.main(CheckReassignmentStatus.scala)

        Activity

        Hide
        Swapnil Ghike added a comment -

        We can make this a sub-task of KAFKA-822 if this does not need to be urgently fixed.

        Show
        Swapnil Ghike added a comment - We can make this a sub-task of KAFKA-822 if this does not need to be urgently fixed.
        Hide
        Neha Narkhede added a comment -

        Good suggestion since it will require the same cleanup you intend to do as part of KAFKA-822

        Show
        Neha Narkhede added a comment - Good suggestion since it will require the same cleanup you intend to do as part of KAFKA-822
        Hide
        Jay Kreps added a comment -

        Neha Narkhede Does this bug still exist?

        Show
        Jay Kreps added a comment - Neha Narkhede Does this bug still exist?
        Hide
        Neha Narkhede added a comment -

        Nope, it's fixed. The tool mentions that the reassignment failed for just the non-existent partition and the rest succeed.

        nnarkhed-mn1:kafka-git-idea nnarkhed$ ./bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file new-assignment.json --verify
        
        Status of partition reassignment:
        ERROR: Assigned replicas () don't match the list of replicas for reassignment (0,1) for partition [test,2]
        Reassignment of partition [test,2] failed
        Reassignment of partition [test,0] completed successfully
        Reassignment of partition [test,1] completed successfully
        
        Show
        Neha Narkhede added a comment - Nope, it's fixed. The tool mentions that the reassignment failed for just the non-existent partition and the rest succeed. nnarkhed-mn1:kafka-git-idea nnarkhed$ ./bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file new-assignment.json --verify Status of partition reassignment: ERROR: Assigned replicas () don't match the list of replicas for reassignment (0,1) for partition [test,2] Reassignment of partition [test,2] failed Reassignment of partition [test,0] completed successfully Reassignment of partition [test,1] completed successfully

          People

          • Assignee:
            Swapnil Ghike
            Reporter:
            Scott Clasen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development