[HIVE-20435] Failed Dynamic Partition Insert into insert only table may loose transaction metadata

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0-alpha-2
• Component/s: Transactions
• Target Version/s: 4.0.0
• Release Note: n/a
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 22/Aug/18 00:40
• Updated: 16/Nov/22 13:50
• Resolved: 09/Jan/19 23:30

Description

TxnHandler.enqueueLockWithRetry() has an optimization where it doesn't writ to TXN_COMPONENTS if the write is a dynamic partition insert because it expects to write to this table from addDynamicPartitions().

For insert-only, transactional tables, we create the target dir and start writing to it before addDynamicPartitions() is called. So if a txn is aborted, we may have a delta dir in the partition but no corresponding entry in TXN_COMPONENTS. This means TxnStore.cleanEmptyAbortedTxns() may clean up TXNS entry for the aborted transaction before Compactor removes this delta dir, at which point it looks like committed data.

Streaming API V2 with dynamic partition mode also has this problem.

Full CRUD are currently immune to this since they rely on "move" operation in MoveTask but longer term they should follow the same model as insert-only tables.

Issue Links

• is related to: HIVE-20436 Lock Manager scalability - linear
• is superceded by: HIVE-21052 Make sure transactions get cleaned if they are aborted before addPartitions is called