[KAFKA-2687] Add support for ListGroups and DescribeGroup APIs

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.9.0.0
• Assignee: Jason Gustafson
• Reporter: Jason Gustafson
• Reviewer: Jun Rao
• Created: 23/Oct/15 19:16
• Updated: 08/Dec/15 17:07
• Resolved: 03/Nov/15 22:40

Description

Since the new consumer currently has no persistence in Zookeeper (pending outcome of KAFKA-2017), there is no way for administrators to investigate group status including getting the list of members in the group and their partition assignments. We therefore propose to modify GroupMetadataRequest (previously known as ConsumerMetadataRequest) to return group metadata when received by the respective group's coordinator. When received by another broker, the request will be handled as before: by only returning coordinator host and port information.

GroupMetadataRequest => GroupId IncludeMetadata
  GroupId => String
  IncludeMetadata => Boolean

GroupMetadataResponse => ErrorCode Coordinator GroupMetadata
  ErrorCode => int16
  Coordinator => Id Host Port
    Id => int32
    Host => string
    Port => int32
  GroupMetadata => State ProtocolType Generation Protocol Leader  Members
    State => String
    ProtocolType => String
    Generation => int32
    Protocol => String
    Leader => String
    Members => [Member MemberMetadata MemberAssignment]
      Member => MemberIp ClientId
        MemberIp => String
        ClientId => String
      MemberMetadata => Bytes
      MemberAssignment => Bytes

The request schema includes a flag to indicate whether metadata is needed, which saves clients from having to read all group metadata when they are just trying to find the coordinator. This is important to reduce group overhead for use cases which involve a large number of topic subscriptions (e.g. mirror maker).

Tools will use the protocol type to determine how to parse metadata. For example, when the protocolType is "consumer", the tool can use ConsumerProtocol to parse the member metadata as topic subscriptions and partition assignments.

The detailed proposal can be found below.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-40%3A+ListGroups+and+DescribeGroup

Issue Links

• is related to: KAFKA-2064 Replace ConsumerMetadataRequest and Response with org.apache.kafka.common.requests objects
• is related to: KAFKA-2901 Extend ListGroups and DescribeGroup APIs to cover offsets