[KAFKA-15178] Poor performance of ConsumerCoordinator with many TopicPartitions

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.5.0
• Component/s: consumer
• Labels: easyfix patch-available
• Assignee: Nicholas Telford
• Reporter: Nicholas Telford
• Created: 11/Jul/23 12:53
• Updated: 07/Dec/23 03:33

Description

Doing some profiling of my Kafka Streams application, I noticed that the pollPhase suffers from a minor performance issue.

See the pink tree on the left of the flame graph below.

ConsumerCoordinator.poll calls rejoinNeededOrPending, which checks the current metadataSnapshot against the assignmentSnapshot. This comparison is a deep-equality check, and if there's a large number of topic-partitions being consumed by the application, then this comparison can perform poorly.

I suspect this can be trivially addressed with a boolean flag that indicates when the metadataSnapshot has been updated (or is "dirty"), and actually needs to be checked, since most of the time it should be identical to assignmentSnapshot.

I plan to raise a PR with this optimization to address this issue.

Issue Links

• links to: GitHub Pull Request #13993