[KAFKA-15539] Client should stop fetching while partitions being revoked

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 3.7.0
• Component/s: clients, consumer
• Labels: kip-848 kip-848-client-support kip-848-preview
• Assignee: Lianet Magrans
• Reporter: Lianet Magrans
• Created: 04/Oct/23 13:27
• Updated: 20/Feb/24 20:44
• Resolved: 30/Oct/23 16:59

Description

When partitions are being revoked (client received revocation on heartbeat and is in the process of invoking the callback), we need to make sure we do not fetch from those partitions anymore:

• no new fetches should be sent out for the partitions being revoked
• no fetch responses should be handled for those partitions (case where a fetch was already in-flight when the partition revocation started.

This does not seem to be handled in the current KafkaConsumer and the old consumer protocol (only for the EAGER protocol). 

Consider re-using the existing pendingRevocation logic that already exist in the subscriptionState & used from the fetcher to determine if a partition is fetchable. 

Issue Links

• is part of: KAFKA-15574 Update states and transitions for membership manager state machine
• is part of: KAFKA-14246 Update threading model for Consumer