[HIVE-15202] Concurrent compactions for the same partition may generate malformed folder structure

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Transactions
• Assignee: Eugene Koifman
• Reporter: Rui Li
• Created: 15/Nov/16 06:52
• Updated: 26/Jul/17 00:03
• Resolved: 01/Dec/16 01:22

Description

If two compactions run concurrently on a single partition, it may generate folder structure like this: (nested base dir)

drwxr-xr-x   - root supergroup          0 2016-11-14 22:23 /user/hive/warehouse/test/z=1/base_0000007/base_0000007
-rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00000
-rw-r--r--   3 root supergroup        611 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00001
-rw-r--r--   3 root supergroup        614 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00002
-rw-r--r--   3 root supergroup        621 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00003
-rw-r--r--   3 root supergroup        621 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00004
-rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00005
-rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00006
-rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00007
-rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00008
-rw-r--r--   3 root supergroup        201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00009

Issue Links

• relates to: HIVE-14980 Minor compaction when triggered simultaniously on the same table/partition deletes data