Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8179 Incremental Rebalance Protocol for Kafka Consumer
  3. KAFKA-8421

Allow consumer.poll() to return data in the middle of rebalance

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.5.0
    • consumer, streams
    • None

    Description

      With KIP-429 in place, today when a consumer is about to send join-group request its owned partitions may not be empty, meaning that some of its fetched data can still be returned. Nevertheless, today the logic is strict:

                          if (!updateAssignmentMetadataIfNeeded(timer)) {
                              return ConsumerRecords.empty();
                          }
      

      I.e. if the consumer enters a rebalance it always returns no data.

      As an optimization, we can consider letting consumers to still return messages that still belong to its owned partitions even when it is within a rebalance, because we know it is safe that no one else would claim those partitions in this rebalance yet, and we can still commit offsets if, after this rebalance, the partitions need to be revoked then.

      One thing we need to take care though is the rebalance timeout, i.e. when consumer's processing those records they may not call the next poll() in time (think: Kafka Streams num.iterations mechanism), which may leads to consumer dropping out of the group during rebalance.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            guozhang Guozhang Wang
            guozhang Guozhang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment