= Intro = The goal of following testing is to be sure that implemented reassigning algorithm works as defined, i.e.: - produces fair assignment (every broker gets X or X+1 replicas); - minimizes number of replica reassignments (moves); = Environment = To accomplish testing following environment is used: - 5 broker nodes in a cluster running inside vagrant boxes; - a topic with 10 partitions / 2 replicas; Topic is created by running: ./kafka-topics.sh --create --topic t --partitions 10 --replication-factor 2 --zookeeper b0:2181 Initial replica distribution: ./kafka-topics.sh --describe --topic t --zookeeper b0:2181Topic:t PartitionCount:10 ReplicationFactor:2 Configs: Topic: t Partition: 0 Leader: 0 Replicas: 0,2 Isr: 0,2 Topic: t Partition: 1 Leader: 1 Replicas: 1,3 Isr: 1,3 Topic: t Partition: 2 Leader: 2 Replicas: 2,4 Isr: 2,4 Topic: t Partition: 3 Leader: 3 Replicas: 3,0 Isr: 3,0 Topic: t Partition: 4 Leader: 4 Replicas: 4,1 Isr: 4,1 Topic: t Partition: 5 Leader: 0 Replicas: 0,3 Isr: 0,3 Topic: t Partition: 6 Leader: 1 Replicas: 1,4 Isr: 1,4 Topic: t Partition: 7 Leader: 2 Replicas: 2,0 Isr: 2,0 Topic: t Partition: 8 Leader: 3 Replicas: 3,1 Isr: 3,1 Topic: t Partition: 9 Leader: 4 Replicas: 4,2 Isr: 4,2 = Tests = All tests are done using kafka-reassign-partitions.sh tool. Tests are splitted into 2 groups: - tests for --generate sub-command; - tests for --decommission-broker sub-command (which uses the same algorithm inside); Following are tests for --generate: 1. reduce broker list from 5 to 4 brokers; 2. expand broker list from 4 to 5 brokers; 3. reduce broker list from 5 to 3 brokers; 4. expand broker list from 3 to 5 brokers; 5. replace single broker with a new one; 6. replace two brokers with new ones; Following are tests for --decommission-broker: 1. decommission specified broker; = reduce broker list from 5 to 4 brokers = ./kafka-reassign-partitions.sh --generate --topics-to-move-json-file topics.json --broker-list 0,1,2,3 --zookeeper b0:2181 Current partition replica assignment {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[2,0]},{"topic":"t","partition":8,"replicas":[3,1]},{"topic":"t","partition":3,"replicas":[3,0]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":1,"replicas":[1,3]},{"topic":"t","partition":6,"replicas":[1,4]},{"topic":"t","partition":2,"replicas":[2,4]},{"topic":"t","partition":4,"replicas":[4,1]},{"topic":"t","partition":9,"replicas":[4,2]},{"topic":"t","partition":5,"replicas":[0,3]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":3,"replicas":[0,3]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":5,"replicas":[0,3]},{"topic":"t","partition":9,"replicas":[0,2]},{"topic":"t","partition":8,"replicas":[1,3]},{"topic":"t","partition":1,"replicas":[1,3]},{"topic":"t","partition":6,"replicas":[1,3]},{"topic":"t","partition":4,"replicas":[1,2]},{"topic":"t","partition":2,"replicas":[1,2]}]} Input: brokers 0 1 2 3 4 parts 0 1 0 1 2 3 4 2 3 4 5 6 7 5 6 7 8 9 8 9 Output: brokers 0 1 2 3 4 parts 0 1 0 1 3 2 2 3 5 4 4 5 7 6 7 6 9 8 9 8 Additions: broker 0 1 2 3 4 parts 9 2 4 6 4 moves were done. = expand broker list from 4 to 5 brokers = ./kafka-reassign-partitions.sh --generate --topics-to-move-json-file topics.json --broker-list 0,1,2,3,4 --zookeeper b0:2181 Current partition replica assignment {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":8,"replicas":[1,3]},{"topic":"t","partition":3,"replicas":[0,3]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":1,"replicas":[1,3]},{"topic":"t","partition":6,"replicas":[1,3]},{"topic":"t","partition":2,"replicas":[1,2]},{"topic":"t","partition":4,"replicas":[1,2]},{"topic":"t","partition":9,"replicas":[0,2]},{"topic":"t","partition":5,"replicas":[0,3]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":3,"replicas":[0,3]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":9,"replicas":[0,4]},{"topic":"t","partition":8,"replicas":[1,3]},{"topic":"t","partition":1,"replicas":[1,3]},{"topic":"t","partition":6,"replicas":[1,4]},{"topic":"t","partition":2,"replicas":[1,2]},{"topic":"t","partition":4,"replicas":[2,4]},{"topic":"t","partition":5,"replicas":[3,4]}]} Input: brokers 0 1 2 3 4 parts 0 1 0 1 3 2 2 3 5 4 4 5 7 6 7 6 9 8 9 8 Output: brokers 0 1 2 3 4 parts 0 1 0 1 4 3 2 2 3 5 7 6 4 5 6 9 8 7 8 9 Removals: broker 0 1 2 3 4 parts 5 4 9 6 4 moves were done. = reduce broker list from 5 to 3 brokers = ./kafka-reassign-partitions.sh --generate --topics-to-move-json-file topics.json --broker-list 0,1,2 --zookeeper b0:2181 Current partition replica assignment {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":8,"replicas":[1,3]},{"topic":"t","partition":3,"replicas":[0,3]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":1,"replicas":[1,3]},{"topic":"t","partition":6,"replicas":[1,4]},{"topic":"t","partition":2,"replicas":[1,2]},{"topic":"t","partition":4,"replicas":[2,4]},{"topic":"t","partition":9,"replicas":[0,4]},{"topic":"t","partition":5,"replicas":[3,4]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":3,"replicas":[0,1]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":9,"replicas":[0,1]},{"topic":"t","partition":8,"replicas":[0,1]},{"topic":"t","partition":1,"replicas":[0,1]},{"topic":"t","partition":4,"replicas":[0,2]},{"topic":"t","partition":6,"replicas":[1,2]},{"topic":"t","partition":2,"replicas":[1,2]},{"topic":"t","partition":5,"replicas":[1,2]}]} Input: brokers 0 1 2 3 4 parts 0 1 0 1 4 3 2 2 3 5 7 6 4 5 6 9 8 7 8 9 Output: brokers 0 1 2 3 4 parts 0 1 0 1 2 2 3 3 4 4 5 5 7 6 6 8 8 7 9 9 Additions: broker 0 1 2 3 4 parts 1 3 5 4 5 6 8 9 8 moves were done. = expand broker list from 3 to 5 brokers = ./kafka-reassign-partitions.sh --generate --topics-to-move-json-file topics.json --broker-list 0,1,2,3,4 --zookeeper b0:2181 Current partition replica assignment {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":8,"replicas":[0,1]},{"topic":"t","partition":3,"replicas":[0,1]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":1,"replicas":[0,1]},{"topic":"t","partition":6,"replicas":[1,2]},{"topic":"t","partition":2,"replicas":[1,2]},{"topic":"t","partition":4,"replicas":[0,2]},{"topic":"t","partition":9,"replicas":[0,1]},{"topic":"t","partition":5,"replicas":[1,2]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":8,"replicas":[0,1]},{"topic":"t","partition":3,"replicas":[0,1]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":1,"replicas":[1,3]},{"topic":"t","partition":6,"replicas":[1,2]},{"topic":"t","partition":2,"replicas":[2,4]},{"topic":"t","partition":4,"replicas":[3,4]},{"topic":"t","partition":9,"replicas":[3,4]},{"topic":"t","partition":5,"replicas":[3,4]}]} Input: brokers 0 1 2 3 4 parts 0 1 0 1 2 2 3 3 4 4 5 5 7 6 6 8 8 7 9 9 Output: brokers 0 1 2 3 4 parts 0 1 0 1 2 3 3 2 4 4 7 6 6 5 5 8 8 7 9 9 Removals: brokers 0 1 2 3 4 parts 1 2 4 4 5 5 9 9 8 moves were done. = replace single broker with a new one = ./kafka-reassign-partitions.sh --generate --topics-to-move-json-file topics.json --broker-list 0,1,2,4 --zookeeper b0:2181 Current partition replica assignment {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":8,"replicas":[1,3]},{"topic":"t","partition":3,"replicas":[0,3]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":1,"replicas":[1,3]},{"topic":"t","partition":6,"replicas":[1,3]},{"topic":"t","partition":2,"replicas":[1,2]},{"topic":"t","partition":4,"replicas":[1,2]},{"topic":"t","partition":9,"replicas":[0,2]},{"topic":"t","partition":5,"replicas":[0,3]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":3,"replicas":[0,4]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":9,"replicas":[0,2]},{"topic":"t","partition":5,"replicas":[0,4]},{"topic":"t","partition":8,"replicas":[1,4]},{"topic":"t","partition":1,"replicas":[1,4]},{"topic":"t","partition":6,"replicas":[1,4]},{"topic":"t","partition":2,"replicas":[1,2]},{"topic":"t","partition":4,"replicas":[1,2]}]} Input: brokers 0 1 2 3 4 parts 0 1 0 1 3 2 2 3 5 4 4 5 7 6 7 6 9 8 9 8 Output: brokers 0 1 2 3 4 parts 0 1 0 1 3 2 2 3 7 4 4 5 5 6 7 6 9 8 9 8 5 moves were done (offload 3 -> 4). = replace two brokers with new ones = ./kafka-reassign-partitions.sh --generate --topics-to-move-json-file topics.json --broker-list 0,3,4 --zookeeper b0:2181 Current partition replica assignment {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,2]},{"topic":"t","partition":8,"replicas":[0,1]},{"topic":"t","partition":3,"replicas":[0,1]},{"topic":"t","partition":0,"replicas":[0,2]},{"topic":"t","partition":1,"replicas":[0,1]},{"topic":"t","partition":6,"replicas":[1,2]},{"topic":"t","partition":2,"replicas":[1,2]},{"topic":"t","partition":4,"replicas":[0,2]},{"topic":"t","partition":9,"replicas":[0,1]},{"topic":"t","partition":5,"replicas":[1,2]}]} Proposed partition reassignment configuration {"version":1,"partitions":[{"topic":"t","partition":7,"replicas":[0,4]},{"topic":"t","partition":8,"replicas":[0,3]},{"topic":"t","partition":3,"replicas":[0,3]},{"topic":"t","partition":0,"replicas":[0,4]},{"topic":"t","partition":1,"replicas":[0,3]},{"topic":"t","partition":4,"replicas":[0,4]},{"topic":"t","partition":9,"replicas":[0,3]},{"topic":"t","partition":6,"replicas":[3,4]},{"topic":"t","partition":2,"replicas":[3,4]},{"topic":"t","partition":5,"replicas":[3,4]}]} Input: brokers 0 1 2 3 4 parts 0 1 0 1 2 2 3 3 4 4 5 5 7 6 6 8 8 7 9 9 Output: brokers 0 1 2 3 4 parts 0 1 0 1 2 2 3 3 4 4 5 5 7 6 6 8 8 7 9 9 13 moves where done (offload (1,2) -> (3,4)) = decommission specified broker = ./kafka-reassign-partitions.sh --decommission-broker 4 --zookeeper b0:2181 Current partition assignment: {"version":1,"partitions":[{"topic":"t","partition":6,"replicas":[1,4]},{"topic":"t","partition":4,"replicas":[2,4]},{"topic":"t","partition":9,"replicas":[0,4]},{"topic":"t","partition":5,"replicas":[3,4]}]} Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"t","partition":9,"replicas":[0,1]},{"topic":"t","partition":5,"replicas":[0,3]},{"topic":"t","partition":6,"replicas":[1,2]},{"topic":"t","partition":4,"replicas":[2,3]}]} Input: brokers 0 1 2 3 4 parts 6 4 5 4 9 5 6 9 Output: brokers 0 1 2 3 4 parts 5 6 4 4 9 9 6 5 Note: during decommission we only take into account replicas for partition assigned to broker being decommissioned. In our case partitions 4,5,6,9 initially had replicas on broker 4.