[KAFKA-17154] New consumer subscribe may join group without a call to consumer.poll

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.8.0
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: kip-848-client-support
• Assignee: PoAn Yang
• Reporter: Lianet Magrans
• Created: 17/Jul/24 14:33
• Updated: 26/Sep/24 20:23
• Resolved: 26/Sep/24 18:55

Description

With the new consumer, a call to consumer.subscribe will update the client subscription state via a SubscriptionChangedEvent. This will be picked up by the background thread, and will send a Heartbeat RPC on the next poll of the background thread (without requiring a consumer.poll). This is a change in behaviour with the legacy consumer, that breaks the consumer#subscribe contract for "rebalances will only occur during an active call to {@link #poll(Duration)}", so it should be fixed.  

In the legacy its a similar principle but different behaviour: subscribe changes the subscription state, and the coordinator picks it up to send the JoinRequest on the next poll of the coordinator (but this happens only as part of a consumer.poll)

We should make the new consumer join (send HB to join) only on consumer.poll after a call to subscribed. We could consider having the SubscriptionChangedEvent only signal to the background that the subscription changed without doing the transition to JOINING that triggers the HB (membershipMgr#onSubscriptionUpdated could simply flip a flag, no transition), and then do the actual transition to JOINING when processing a PollEvent if the flag is on (flipping it back). (Just first simple approach that comes to mind, we should think about it a bit more and consider other interactions that could happen in between, like unsubscribe, close, etc).

We should add test coverage to ensure the common behaviour with the legacy.

Issue Links

• links to: GitHub Pull Request #17165