[HIVE-21266] Don't run cleaner if compaction is skipped (issue with single delta file)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Transactions
• Target Version/s: 4.0.0
• Assignee: Karen Coppage Assign to me
• Reporter: Eugene Koifman
• Created: 13/Feb/19 20:26
• Updated: 17/Nov/22 08:54
• Resolved: 06/Dec/19 10:11

Description

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357

 

if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1) {
      LOG.debug("Not compacting {}; current base is {} and there are {} deltas and {} originals", sd.getLocation(), dir
          .getBaseDirectory(), deltaCount, origCount);
      return;
    }
 

Is problematic.
Suppose you have 1 delta file from streaming ingest: delta_11_20 where txnid:13 was aborted. The code above will not rewrite the delta (which drops anything that belongs to the aborted txn) and transition the compaction to "ready_for_cleaning" state which will drop the metadata about the aborted txn in markCleaned(). Now aborted data will come back as committed.

Issue Links

• relates to: HIVE-9995 ACID compaction tries to compact a single file
• links to: Review board link