Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.8.0, 3.9.0
-
None
-
None
Description
With KAFKA-17105, a scenario is described where a connector may be unnecessarily restarted soon after it has been created.
Similarly, when any events occur that set the DistributedHerder.needsReconfigRebalance flag to true (at the time of writing these are the detection of a new connector, the removal of an existing connector, or the detection of new task configurations regardless of whether existing configurations existed for the connector), it is possible that a rebalance has already started because another worker has detected this change as well. In that case, needsReconfigRebalance will still be set to true even after that rebalance has taken place, and the worker will force an unnecessary second rebalance.
We might consider changing the "needs reconfig rebalance" field into a "reconfig rebalance threshold" field, which contains the latest offset of a record consumed from the config topic that warrants a rebalance. When possibly performing rebalances based on this field, the worker can check if the offset in the assignment given out by the leader during the most recent rebalance is greater than or equal to this threshold, and if so, choose not to force a rebalance.
This has caused issues in some tests, but may be a benign race condition that does not have practical consequences in the real world. We may not want to address this (especially with an approach that increases the complexity of the code base and comes with risk of regression) until/unless someone states that it's affected them outside of Kafka Connect unit tests.