[HBASE-3404] Compaction Ordering for Bulk Import Files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.90.0, 0.90.1, 0.92.0
• Assignee: Nicolas Spiegelberg
• Reporter: Nicolas Spiegelberg
• Created: 31/Dec/10 00:01
• Updated: 12/Jun/22 00:42
• Resolved: 11/Apr/15 00:54

Description

We got into an issue today where we were using HFileOutputFormat to perform an incremental load on an already-large cluster. Because bulk-loaded files don't have a sequence ID, they are put in the front of the StoreFile list. This resulted in the following StoreFile ordering

2GB (bulk) => 25GB => 2GB => ...

So this triggered a 30+GB major compaction for every single region. Optimally, we would like bulk import files to be ordered in the compaction list at the time of insertion so this can be a much smaller compaction and rely on StoreFile age for major compaction trigger.

Issue Links

• is related to: HBASE-3690 Option to Exclude Bulk Import Files from Minor Compaction
• relates to: HBASE-1923 Bulk incremental load into an existing table
• requires: HBASE-2856 TestAcidGuarantee broken on trunk