[KAFKA-4047] Return more useful information from ConsumerGroupCommand for consumer groups that are rebalancing or manually assigned

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.9.0.1, 0.10.0.0, 0.10.0.1
• Component/s: consumer, tools
• Assignee: Unassigned
• Reporter: Sean McKibben
• Created: 16/Aug/16 15:58
• Updated: 28/Jun/17 12:44
• Resolved: 16/Aug/16 18:18

Description

For consumers which are manually assigned topic partitions, or for offsets which are otherwise manually committed to brokers using the new consumer API, the existing ConsumerGroupCommand tools are operationally insufficient.
There are many use cases in production operation when it is important to be able to easily retrieve the stored offsets and calculated lag of a consumer group, even if the group is rebalancing or partition assignment information is indeterminate. Often these cases involve a misbehaving or crashed client application, and having the ConsumerGroupCommand return Consumer group `myGroupID` does not exist or is rebalancing. instead of the information it does know is not very helpful. Additionally, when manual offset commits are used, or the automatic consumer group subscription management is not used, the same message is returned by the tool. In these cases, the offsets are actually stored and available even though the broker/coordinator doesn't have information about partition ownership.
Previously, this was a non-issue as a ZK client could be leveraged relatively easily to get important information like the last stored offset positions, or consumer-offset-checker would return more information in its responses. With the new consumer, however, the process for getting offset and lag is not as straightforward, and tools to provide this information at the operational level are necessary as part of a Kafka installation.

Issue Links

• duplicates: KAFKA-3853 Report offsets for empty groups in ConsumerGroupCommand