Description
There are a few small changes that we might make to the incremental rebalancing logic for Kafka Connect to improve distribution of connectors and tasks across a cluster and address potential bugs:
- During assignment, assign new connectors and tasks across the cluster before calculating revocations that may be necessary in order to balance the cluster. This way, we can potentially skip a round of revocation by using newly-created connectors and tasks to balance out the cluster.
- Perform connector and task revocation in more cases, such as when one or more connectors are reconfigured to use fewer tasks, which can possibly lead to an imbalanced cluster.
- Fix this line to use the same aggregation logic that's used here in order to avoid overwriting map values when they should be combined instead.
Attachments
Issue Links
- relates to
-
KAFKA-12495 Unbalanced connectors/tasks distribution will happen in Connect's incremental cooperative assignor
-
- Resolved
-
- links to