[HIVE-25600] Compaction job creates redundant base/delta folder within base/delta folder

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.2
• Fix Version/s: 3.1.3
• Component/s: Hive
• Labels: ACID Compaction pull-request-available
• Target Version/s: 3.1.3
• Assignee: Nikhil Gupta
• Reporter: Nikhil Gupta
• Created: 07/Oct/21 17:05
• Updated: 09/Apr/22 07:03
• Resolved: 27/Oct/21 08:06

Description

Hive table 'aa.bb_item' is corrupt. Found sub-directory 'abfs://[e-mail]/prod-data/aa.db/bb_item/part_created_on=202105/base_0004042/base_0004042' in bucket directory for partition: part_created_on=202105
 at io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:543)
 at io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:325)
 at io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:254)
 at io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
 at io.prestosql.$gen.Presto_347____20210615_143054_2.run(Unknown Source)
 at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:829);

Why it happens:
Multiple compaction jobs for the same transactions can be triggered if the HMS gets restarted and the MR job is still in progress.

Issue Links

• links to: GitHub Pull Request #2705