[HIVE-12353] When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it should not.

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Transactions
• Labels: TODOC1.3
• Target Version/s: 1.3.0, 2.0.0
• Assignee: Eugene Koifman Assign to me
• Reporter: Eugene Koifman
• Created: 05/Nov/15 21:29
• Updated: 01/Oct/19 22:07
• Resolved: 22/Jan/16 03:02

Description

One of the things that this method does is delete entries from TXN_COMPONENTS for partition that it was trying to compact.
This causes Aborted transactions in TXNS to become empty according to
CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be deleted.
Once they are deleted, data that belongs to these txns is deemed committed...

We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success) states. We should also not delete then entry from markedCleaned()
We'll have separate process that cleans 'f' and 's' records after X minutes (or after > N records for a given partition exist).
This allows SHOW COMPACTIONS to show some history info and how many times compaction failed on a given partition (subject to retention interval) so that we don't have to call markCleaned() on Compactor failures at the same time preventing Compactor to constantly getting stuck on the same bad partition/table.

Ideally we'd want to include END_TIME field.

Issue Links

• blocks: HIVE-11994 SHOW COMPACTIONS should include meta info about the job
• is blocked by: HIVE-12832 RDBMS schema changes for HIVE-11388
• is blocked by: HIVE-12352 CompactionTxnHandler.markCleaned() may delete too much
• is related to: HIVE-12352 CompactionTxnHandler.markCleaned() may delete too much
• is related to: HIVE-13691 No record with CQ_ID=0 found in COMPACTION_QUEUE
• is related to: HIVE-11444 ACID Compactor should generate stats/alerts
• requires: HIVE-13353 SHOW COMPACTIONS should support filtering options

Sub-Tasks

• Schema changes for HIVE-12353: Resolved