[KAFKA-16780] Txn consumer exerts pressure on remote storage when collecting aborted transactions

• Type: Task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Labels: kip
• Assignee: Kamal Chandraprakash
• Reporter: Kamal Chandraprakash
• Created: 16/May/24 08:49
• Updated: 2 hours ago

Description

Logic to read aborted txns:

1. When the consumer enables isolation_level as READ_COMMITTED and reads a non-txn topic, then the broker has to traverse all the local log segments to collect the aborted transactions since there won't be any entry in the transaction index.
2. The same logic is applied while reading from remote storage. In this case, when the FETCH request is reading data from the first remote log segment, then it has to fetch the transaction indexes of all the remaining remote-log segments, and then the call lands to the local-log segments before responding to the FETCH request which increases the time taken to serve the requests.

The EoS Abort Index design doc explains how the transaction index file filters out the aborted transaction records.

The issue is when consumers are enabled with the READ_COMMITTED isolation level but read the non-txn topics. If the topic is enabled with the transaction, then we expect the transaction to either commit/rollback within 15 minutes (default transaction.max.timeout.ms = 15 mins), possibly we may have to search only a few remote log segments to collect the aborted txns.

Issue Links

• is a child of: KAFKA-16947 Kafka Tiered Storage V2
• relates to: KAFKA-15678 [Tiered Storage] Stall remote reads with long-spanning transactions
• links to: GitHub Pull Request #17659
• links to: GitHub Pull Request #17668