[KAFKA-9286] Improve producer performance

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.2.2
• Component/s: clients
• Labels: performance
• Flags: Patch
• External issue URL: https://github.com/apache/kafka/pull/7797
• Assignee: Unassigned
• Reporter: louxiu
• Created: 07/Dec/19 09:40
• Updated: 18/May/20 02:52

Description

Pull request: https://github.com/apache/kafka/pull/7797

For producer, the `Sender` [call](https://github.com/apache/kafka/blob/ba365bbb8deb16ec133e0e7983c1c56cef0152aa/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L342) `RecordAccumulator#ready` to collect all ready nodes. Then it [call](https://github.com/apache/kafka/blob/ba365bbb8deb16ec133e0e7983c1c56cef0152aa/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L369) `RecordAccumulator#drain` to iterate all ready nodes's topic partition to check if there is data to send.

This process is redundancy. When we collect all ready nodes, its ready topic partition can also be collected.

In our case, the producer only produce data to part(about 20) of all partitions(1024), with this patch, the performance can be increased ~3.4%.

Issue Links

• links to: GitHub Pull Request #7797