[HIVE-21052] Make sure transactions get cleaned if they are aborted before addPartitions is called

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.1
• Component/s: Transactions
• Labels: pull-request-available
• Assignee: Jaume M
• Reporter: Jaume M
• Created: 17/Dec/18 23:05
• Updated: 05/Oct/21 08:36
• Resolved: 09/Sep/21 09:24

Description

If the transaction is aborted between openTxn and addPartitions and data has been written on the table the transaction manager will think it's an empty transaction and no cleaning will be done.
This is currently an issue in the streaming API and in micromanaged tables. As proposed by ekoifman this can be solved by:

• Writing an entry with a special marker to TXN_COMPONENTS at openTxn and when addPartitions is called remove this entry from TXN_COMPONENTS and add the corresponding partition entry to TXN_COMPONENTS.
• If the cleaner finds and entry with a special marker in TXN_COMPONENTS that specifies that a transaction was opened and it was aborted it must generate jobs for the worker for every possible partition available.

cc ewohlstadter

Issue Links

• causes: HIVE-25502 Cleaner causes data loss when processing aborted txn with DP
• is related to: HIVE-24374 Do not remove metadata for aborted writes that were batched with valid ones in the same delta file
• is related to: HIVE-21150 Don't block Cleaner at the end of the loop
• relates to: HIVE-21118 Make sure if dynamic partitions is true only there's only one writeId allocated
• supercedes: HIVE-20435 Failed Dynamic Partition Insert into insert only table may loose transaction metadata
• links to: GitHub Pull Request #1415
• links to: GitHub Pull Request #1548
• links to: Review Board