[KAFKA-13420] consumer protocol should include "generation" field for assignor to distinguish between new/old OwnedPartitions

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: clients, consumer
• Labels: need-kip
• Assignee: Luke Chen
• Reporter: Luke Chen
• Created: 29/Oct/21 12:53
• Updated: 01/Nov/21 08:55

Description

In KIP-429, we add a new field: `OwnedPartitions` into consumer protocol, for cooperative protocol do partition revoking things. But recently, we found the `ownedPartitions` info might be out-of-date due to some reasons (ex: unstable network), and the out-of-date  `ownedPartitions` causes unexpected rebalance stuck issue (ex: KAFKA-12984, KAFKA-13406). To fix it, we should consider to add the "generation" field in the consumer protocol, so that we can rely on the "generation" info to identify if the `ownedPartition` is up-to-date or out-of-date.

Issue Links

• is related to: KAFKA-12984 Cooperative sticky assignor can get stuck with invalid SubscriptionState input metadata