[HIVE-20823] Make Compactor run in a transaction

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Transactions
• Target Version/s: 4.0.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 27/Oct/18 00:01
• Updated: 17/Nov/22 08:55
• Resolved: 04/Dec/18 00:06

Description

Have compactor open a transaction and run the job in that transaction.

1. make compactor produced base/delta include this txn id in the folder name, e.g. base_7_c17 where 17 is the txnid.
2. add CQ_TXN_ID bigint to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to record this txn id
3. make sure AcidUtils.getAcidState() pays attention to this transaction on read and ignores this dir if this txn id is not committed in the current snapshot
   1. this means not only validWriteIdList but ValidTxnIdList should be passed along in config (if it isn't yet)
4. once this is done, CompactorMR.createCompactorMarker() can be eliminated and AcidUtils.isValidBase modified accordingly
5. modify Cleaner so that it doesn't clean old files until new file is visible to all readers

Issue Links

• causes: HIVE-21213 Acid table bootstrap replication needs to handle directory created by compaction with txn id
• is depended upon by: HIVE-22755 Cleaner/Compaction can skip the read locks and use the min open txn id
• is required by: HIVE-20960 Make MM compactor run in a transaction and remove CompactorMR.createCompactorMarker()
• relates to: HIVE-20948 Eliminate file rename in compactor
• relates to: HIVE-20901 running compactor when there is nothing to do produces duplicate data
• relates to: HIVE-20699 Query based compactor for full CRUD Acid tables
• supercedes: HIVE-18772 Make Acid Cleaner use MIN_HISTORY_LEVEL
• links to: Review Board