[KAFKA-14055] Transaction markers may be lost during cleaning if data keys conflict with marker keys

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.1, 2.5.1, 2.6.3, 2.7.2, 2.8.1, 3.0.1, 3.2.0, 3.1.1
• Fix Version/s: 3.3.0, 3.0.2, 3.1.2, 3.2.1
• Assignee: Jason Gustafson
• Reporter: Jason Gustafson
• Created: 07/Jul/22 19:41
• Updated: 10/Jul/22 17:37
• Resolved: 10/Jul/22 17:36

Description

We have been seeing recently hanging transactions occur on streams changelog topics quite frequently. After investigation, we found that the keys used in the changelog topic conflict with the keys used in the transaction markers (the schema used in control records is 4 bytes, which happens to be the same for the changelog topics that we investigated). When we build the offset map prior to cleaning, we do properly exclude the transaction marker keys, but the bug is the fact that we do not exclude them during the cleaning phase. This can result in the marker being removed from the cleaned log before the corresponding data is removed when there is a user record with a conflicting key at a higher offset. A side effect of this is a so-called "hanging" transaction, but the bigger problem is that we lose the atomicity of the transaction. 

Issue Links

• links to: GitHub Pull Request #12390