[KAFKA-7854] Behavior change in controller picking up partition reassignment tasks since 1.1.0

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: controller
• Assignee: Unassigned
• Reporter: Zhanxiang (Patrick) Huang
• Created: 22/Jan/19 17:34
• Updated: 28/Jan/20 16:33
• Resolved: 28/Jan/20 16:33

Description

After https://github.com/apache/kafka/pull/4143, the controller does not subscribe to data change on /admin/reassign_partitions any more (in order to avoid unnecessarily loading the reassignment data again after controller updating the znode) as opposed to the previous kafka versions. However, there are systems built around kafka relying on the previous behavior to incrementally update the list of partition reassignment since kafka does not natively support that.

 

For example, cruise control can rely on the previous behavior (controller listening to data changes) to maintain the reassignment concurrency by dynamically updating the data in the reassignment znode instead of waiting for the current batch to finish and doing reassignment batch by batch, which can significantly reduce the rebalance time in production clusters. Although directly updating the znode can somehow be viewed as an anti-pattern in the long term, this is necessary since kafka does not natively support incrementally submit more reassignment tasks. However, after our kafka clusters migrate from 0.11 to 2.0, cruise control no longer works because the controller behavior has changed. This reveals the following problems:

• These behavior changes may be viewed as internal changes so compatibility is not guaranteed but I think by convention people do view this as public interfaces and rely on the compatibility. In this case, I think we should clearly document the data contract for the partition reassignment task to avoid misusage and making controller changes that break the defined data contract. There may be other cases (e.g. topic deletion) whose data contracts need to be clearly defined and we should keep it in mind when making controller changes.
• Kafka does not natively support incrementally submit more reassignment tasks. If we do want to support that nicely, we should consider change how we store the reassignment data to store the data in child nodes and let the controller listen on child node changes, similar to what we do for /admin/delete_topics.