Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14048 The Next Generation of the Consumer Rebalance Protocol
  3. KAFKA-15276

Implement event plumbing for ConsumerRebalanceListener callbacks

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Provide the Java client support for the consumer group partition assignment logic, including:

      • Calculate the difference between the current partition assignment and that returned in the ConsumerGroupHeartbeatResponse RPC response
      • Ensure we handle the case where changes to the assignment take multiple passes of RequestManager.poll()
      • Integrate the mechanism to invoke the user’s rebalance callback

      This task is part of the work to implement support for the new KIP-848 consumer group protocol.

      One challenge to solve here is that callback execution, when triggered from the unsubscribe, would require the user to poll for complete the callbacks and then return in the unsubscribe (legacy unubscribe does not require user poll). We might need to re-think callbacks execution considering that they do need to be executed in the app thread, but should not require the user to poll (seeing that they are not events directly generated by the user. They are generated internally by the background thread, to be executed in the app thread) 

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kirktrue Kirk True
            kirktrue Kirk True
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment