Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3949

Consumer topic subscription change may be ignored if a rebalance is in progress

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.1, 0.10.0.0
    • Fix Version/s: 0.10.1.0
    • Component/s: consumer
    • Labels:
      None

      Description

      The consumer's regex subscription works by matching all topics fetched from a metadata update against the provided pattern. When a new topic is created or an old topic is deleted, we update the list of subscribed topics and request a rebalance by setting the needsPartitionAssignment flag inside SubscriptionState. On the next call to poll(), the consumer will observe the flag and begin the rebalance by sending a JoinGroup. The problem is that it does not account for the fact that a rebalance could already be in progress at the time the metadata is updated. This causes the following sequence:

      1. Rebalance begins (needsPartitionAssignment is set True)
      2. Metadata max age expires and and update is triggered
      3. Update returns and causes a topic subscription change (needsPartitionAssignment set again to True).
      4. Rebalance completes (needsPartitionAssignment is set False)

      In this situation, we will not request a new rebalance which will prevent us from receiving an assignment from any topics added to the consumer's subscription when the metadata was updated. This state will persist until another event causes the group to rebalance.

      A related problem may occur if a rebalance is interrupted with the wakeup() API, and the user calls subscribe(topics) with a change to the subscription set.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                hachikuji Jason Gustafson
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: