[KAFKA-13764] Potential improvements for Connect incremental rebalancing logic

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.4.0
• Component/s: connect
• Assignee: Chris Egerton
• Reporter: Chris Egerton
• Created: 23/Mar/22 21:31
• Updated: 15/Nov/22 21:28
• Resolved: 15/Nov/22 21:27

Description

There are a few small changes that we might make to the incremental rebalancing logic for Kafka Connect to improve distribution of connectors and tasks across a cluster and address potential bugs:

1. During assignment, assign new connectors and tasks across the cluster before calculating revocations that may be necessary in order to balance the cluster. This way, we can potentially skip a round of revocation by using newly-created connectors and tasks to balance out the cluster.
2. Perform connector and task revocation in more cases, such as when one or more connectors are reconfigured to use fewer tasks, which can possibly lead to an imbalanced cluster.
3. Fix this line to use the same aggregation logic that's used here in order to avoid overwriting map values when they should be combined instead.

Issue Links

• relates to: KAFKA-12495 Unbalanced connectors/tasks distribution will happen in Connect's incremental cooperative assignor
• links to: Downstream pull request
• links to: GitHub Pull Request #12019
• links to: GitHub Pull Request #12561