[KAFKA-12983] onJoinPrepare is not always invoked before joining the group

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.7.2, 2.8.1, 3.0.0
• Component/s: consumer
• Labels: new-consumer-threading-should-fix
• Assignee: A. Sophie Blee-Goldman
• Reporter: A. Sophie Blee-Goldman
• Created: 23/Jun/21 01:57
• Updated: 06/Jun/22 21:10
• Resolved: 13/Jul/21 19:29

Description

As the title suggests, the #onJoinPrepare callback is not always invoked before a member (re)joins the group, but only once when it first enters the rebalance. This means that any updates or events that occur during the join phase can be lost in the internal state: for example, clearing the SubscriptionState (and thus the "ownedPartitions" that are used for cooperative rebalancing) after losing its memberId during a rebalance.

We should reset the `needsJoinPrepare` flag inside the resetStateAndRejoin() method

Issue Links

• causes: KAFKA-12984 Cooperative sticky assignor can get stuck with invalid SubscriptionState input metadata
• fixes: KAFKA-12896 Group rebalance loop caused by repeated group leader JoinGroups
• supercedes: KAFKA-12920 Consumer's cooperative sticky assignor need to clear generation / assignment data upon `onPartitionsLost`
• links to: GitHub Pull Request #10986
• links to: GitHub Pull Request #11332