[KAFKA-8788] Optimize client metadata handling with a large number of partitions

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0, 2.3.1
• Assignee: Ismael Juma
• Reporter: Ismael Juma
• Created: 11/Aug/19 06:39
• Updated: 15/Aug/19 02:49
• Resolved: 15/Aug/19 02:49

Description

Credit to lbradstreet for profiling the producer with a large number of partitions.

There was a regression in 2.3.0 that made the client metadata handling significantly less efficient when dealing with a large number of partitions. There was a less significant regression in 2.2.0.

Lucas benchmarked the producer, but the code is shared with the consumer. See the PR for more details.

 

Issue Links

• links to: GitHub Pull Request #7192