[HBASE-8765] split should be based on store size, not HFile size

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 0.95.1
• Assignee: Unassigned
• Reporter: Sergey Shelukhin
• Created: 19/Jun/13 00:12
• Updated: 16/Jun/22 17:12
• Resolved: 19/Jun/13 00:16

Description

I noticed that the current split behavior is rather suboptimal with regard to compactions. On large regions, HFile size limit triggers a split. Split is followed by major compaction to get rid of the partial reference files. However, HFile size limit is surpassed after compaction most of the time.
So, first we rewrite a lot of data into a new file. Then we say "Oh look! A large file!", split the region and rewrite everything again.

Perhaps region split should be based on store size, or incoming compaction size - large enough compaction should be converted into splits.

Thoughts? I think basing off store size is a simple fix, and will code it up soon if there are no objections