Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3854

Subsequent regex subscription calls fail

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.10.0.1, 0.10.1.0
    • consumer
    • None

    Description

      There are a couple of issues with regex subscription in the new consumer:

      Subsequent Subscriptions Fail

      When consecutive calls are made to new consumer's regex subscription, like below:

      consumer.subscribe(Pattern.compile("..."), listener);
      consumer.poll(0);
      
      consumer.subscribe(Pattern.compile("f.."), listener);
      consumer.poll(0);
      

      the second call fails with the following error:

      Exception in thread "main" java.lang.IllegalStateException: Subscription to topics, partitions and pattern are mutually exclusive
      	at org.apache.kafka.clients.consumer.internals.SubscriptionState.subscribe(SubscriptionState.java:175)
      	at org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(KafkaConsumer.java:854)
      	...
      
      Subsequent Subscriptions Fail to Trigger a Subscription Change and Rebalance

      Even after the code is tweaked to get around the above issue, only the first call to regex subscription triggers an actual subscription and the subscription's rebalance listener. The reason is the regex subscription / poll does not directly call changeSubscription(...) in which needsPartitionAssignment is set and causes a rebalance. This method is called only during the first regex subscription / poll when coordinator is unknown and a client.poll call is made which eventually leads to a changeSubscription call. The subsequent call does not reach this point because the coordinator is already known.

      It seems due to the same reason, if a consumer is subscribed to a pattern, and later on a new topic is created that matches that pattern, the consumer does not become subscribed it.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vahid Vahid Hashemian
            vahid Vahid Hashemian
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment