[KAFKA-12896] Group rebalance loop caused by repeated group leader JoinGroups

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 3.0.0
• Component/s: clients
• Assignee: David Jacot
• Reporter: Lucas Bradstreet
• Created: 04/Jun/21 14:14
• Updated: 15/Jul/21 07:33
• Resolved: 15/Jul/21 07:33

Description

We encountered a strange case of a rebalance loop with the "cooperative-sticky" assignor. The logs show the following for several hours:

 

{{Apr 7, 2021 @ 03:58:36.040 [GroupCoordinator 7]: Stabilized group mygroup generation 19830137 (__consumer_offsets-7)}}

{{Apr 7, 2021 @ 03:58:35.992 [GroupCoordinator 7]: Preparing to rebalance group mygroup in state PreparingRebalance with old generation 19830136 (__consumer_offsets-7) (reason: Updating metadata for member mygroup-1-7ad27e07-3784-4588-97e1-d796a74d4ecc during CompletingRebalance)}}

{{Apr 7, 2021 @ 03:58:35.988 [GroupCoordinator 7]: Stabilized group mygroup generation 19830136 (__consumer_offsets-7)}}

{{Apr 7, 2021 @ 03:58:35.972 [GroupCoordinator 7]: Preparing to rebalance group mygroup in state PreparingRebalance with old generation 19830135 (__consumer_offsets-7) (reason: Updating metadata for member mygroup during CompletingRebalance)}}

{{Apr 7, 2021 @ 03:58:35.965 [GroupCoordinator 7]: Stabilized group mygroup generation 19830135 (__consumer_offsets-7)}}

{{Apr 7, 2021 @ 03:58:35.953 [GroupCoordinator 7]: Preparing to rebalance group mygroup in state PreparingRebalance with old generation 19830134 (__consumer_offsets-7) (reason: Updating metadata for member mygroup-7ad27e07-3784-4588-97e1-d796a74d4ecc during CompletingRebalance)}}

{{Apr 7, 2021 @ 03:58:35.941 [GroupCoordinator 7]: Stabilized group mygroup generation 19830134 (__consumer_offsets-7)}}

{{Apr 7, 2021 @ 03:58:35.926 [GroupCoordinator 7]: Preparing to rebalance group mygroup in state PreparingRebalance with old generation 19830133 (__consumer_offsets-7) (reason: Updating metadata for member mygroup during CompletingRebalance)}}

Every single time, it was the same member that triggered the JoinGroup and it was always the leader of the group.{{}}

The leader has the privilege of being able to trigger a rebalance by sending `JoinGroup` even if its subscription metadata has not changed. But why would it do so?

It is possible that this is due to the same issue or a similar bug to https://issues.apache.org/jira/browse/KAFKA-12890.

Issue Links

• is caused by: KAFKA-12984 Cooperative sticky assignor can get stuck with invalid SubscriptionState input metadata
• is fixed by: KAFKA-12983 onJoinPrepare is not always invoked before joining the group
• is fixed by: KAFKA-12984 Cooperative sticky assignor can get stuck with invalid SubscriptionState input metadata