[KAFKA-17233] MirrorCheckpointConnector should use batched listConsumerGroupOffsets

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: mirrormaker
• Labels: newbie
• Assignee: João Pedro Fonseca
• Reporter: Greg Harris
• Created: 31/Jul/24 19:44
• Updated: 24/Oct/24 07:39
• Resolved: 24/Oct/24 07:39

Description

The AdminClient provides us a single-group listConsumerGroupOffsets, and a batched listConsumerGroupOffsets. Because we are potentially requesting the offsets for hundreds or thousands of groups, serially requesting all of the offsets may be very slow.

Issue Links

• is related to: KAFKA-17232 MirrorCheckpointConnector does not generate task configs if initial consumer group load times out
• is related to: KAFKA-14837 The MirrorCheckPointConnector of MM2 will rebalance frequently, when the source cluster group is many more and changes frequently (but the list of configured synchronous group does not change)
• links to: GitHub Pull Request #17038