[KAFKA-2901] Extend ListGroups and DescribeGroup APIs to cover offsets

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.9.0.0
• Component/s: consumer
• Assignee: Jason Gustafson
• Reporter: Andy Coates
• Created: 27/Nov/15 11:12
• Updated: 07/Oct/16 16:42
• Resolved: 13/Sep/16 00:12

Description

The ListGroupsRequest and DescribeGroupsRequest added to 0.9.0.0 allow admin tools to get details of consumer groups now that this information is not stored in ZK.

The brokers also now store offset information for consumer groups. At present, there is no API for admin tools to discover the groups that brokers are storing offsets for.

For example, if a consumer is using the new consumer api, is storing offsets in Kafka under the groupId 'Bob', but is using manual partition assignment, then the groupCache in the GroupMetadataManager will not contain any information about the group 'Bob'. However, the offsetCache in the GroupMetadataManager will contain information about 'Bob'.

Currently the only way for admin tools to know the full set of groups being managed by Kafka, i.e. those storing offsets in Kafka, those using Kafka for balancing of consumer groups, and those using Kafka for both, is to consume the offset topic.

We need to extend the List/Describe groups API to allow admin tools to discover 'Bob' and allow the partition offsets to be retrieved.

Issue Links

• duplicates: KAFKA-3853 Report offsets for empty groups in ConsumerGroupCommand
• relates to: KAFKA-2687 Add support for ListGroups and DescribeGroup APIs