Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Do
-
None
-
None
-
None
-
None
Description
Currently controller will update partition reassignment data every time a partition in the reassignment is completed. It means that if user specifies a huge reassignment znode of size 1 MB to move 10K partitions, controller will need to write roughly 0.5 MB * 10000 = 5 GB data to zookeeper in order to complete this reassignment. This is because controller needs to write the remaining partitions to the znode every time a partition is completely moved.
This is problematic because such a huge reassignment may greatly slow down Kafka controller. Note that partition reassignment doesn't necessarily cause data movement between brokers because we may use it only to recorder the replica list of partitions to evenly distribute preferred leader.