[HBASE-4717] More efficient age-off of old data during major compaction

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 0.94.0
• Component/s: regionserver
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 01/Nov/11 17:42
• Updated: 12/Jun/22 19:48
• Resolved: 11/Apr/15 01:16

Description

Many applications need to implement efficient age-off of old data. We currently only perform age-off during major compaction by scanning through all of the KVs. Instead, we could implement the following:

• Set hbase.hstore.compaction.max.size reasonably small. Thus, older store files contain only smaller finite ranges of time.
• Periodically run an "age-off compaction". This compaction would scan the current list of storefiles. Any store file that falls entirely out of the TTL time range would be dropped. Store files completely within the time range would be un-altered. Those crossing the time-range boundary could either be left alone or compacted using the existing compaction code.

I don't have a design in mind for how exactly this would be implemented, but hope to generate some discussion.

Issue Links

• is related to: HBASE-5199 Delete out of TTL store files before compaction selection
• is related to: HBASE-5274 Filter out the expired store file scanner during the compaction