[KAFKA-8338] Improve consumer offset expiration logic to take subscription into account

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: huxihx
• Reporter: Gwen Shapira
• Created: 08/May/19 18:47
• Updated: 02/May/22 18:38
• Resolved: 03/Jun/20 16:31

Description

Currently, we expire consumer offsets for a group after the group is considered gone.

There is a case where the consumer group still exists, but is now subscribed to different topics. In that case, the offsets of the old topics will never expire and if lag is monitored, the monitors will show ever-growing lag on those topics.

We need to improve the logic to expire the consumer offsets if the consumer group didn't subscribe to specific topics/partitions for enough time.

Issue Links

• causes: KAFKA-13636 Committed offsets could be deleted during a rebalance if a group did not commit for a while
• is duplicated by: KAFKA-8730 Add API to delete consumer offsets (KIP-496)
• links to: GitHub Pull Request #6737