[HIVE-8966] Delta files created by hive hcatalog streaming cannot be compacted

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 1.0.0
• Component/s: HCatalog

Environment

hive

• Release Note: Don't do compaction on the current delta if it has a file in bucket pattern but not compactable
• Assignee: Alan Gates
• Reporter: Jihong Liu
• Created: 25/Nov/14 20:32
• Updated: 19/Feb/15 18:22
• Resolved: 27/Jan/15 19:19

Description

hive hcatalog streaming will also create a file like bucket_n_flush_length in each delta directory. Where "n" is the bucket number. But the compactor.CompactorMR think this file also needs to compact. However this file of course cannot be compacted, so compactor.CompactorMR will not continue to do the compaction.

Did a test, after removed the bucket_n_flush_length file, then the "alter table partition compact" finished successfully. If don't delete that file, nothing will be compacted.
This is probably a very severity bug. Both 0.13 and 0.14 have this issue