Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.9.0.1, 0.10.0.0
Description
The consumer's regex subscription works by matching all topics fetched from a metadata update against the provided pattern. When a new topic is created or an old topic is deleted, we update the list of subscribed topics and request a rebalance by setting the needsPartitionAssignment flag inside SubscriptionState. On the next call to poll(), the consumer will observe the flag and begin the rebalance by sending a JoinGroup. The problem is that it does not account for the fact that a rebalance could already be in progress at the time the metadata is updated. This causes the following sequence:
1. Rebalance begins (needsPartitionAssignment is set True)
2. Metadata max age expires and and update is triggered
3. Update returns and causes a topic subscription change (needsPartitionAssignment set again to True).
4. Rebalance completes (needsPartitionAssignment is set False)
In this situation, we will not request a new rebalance which will prevent us from receiving an assignment from any topics added to the consumer's subscription when the metadata was updated. This state will persist until another event causes the group to rebalance.
A related problem may occur if a rebalance is interrupted with the wakeup() API, and the user calls subscribe(topics) with a change to the subscription set.
Attachments
Issue Links
- links to
