[KAFKA-10702] Slow replication of empty transactions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0
• Assignee: Jason Gustafson
• Reporter: Jason Gustafson
• Created: 09/Nov/20 18:35
• Updated: 30/Nov/20 22:49
• Resolved: 30/Nov/20 22:49

Description

We hit a case in which we had to re-replicate a compacted topic from the beginning of the log. Some portions of the log consisted mostly of transaction markers, which were extremely slow to replicate. The problem is that `ProducerStateManager` adds all of these empty transactions to its internal collection of `ongoingTxns` before immediately removing them. There could be tens of thousands of empty transactions in the worst case from a single `Fetch` response, so this can create a huge amount of pressure on the broker.

Issue Links

• links to: GitHub Pull Request #9632