[HBASE-24289] Heterogeneous Storage for Date Tiered Compaction

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.4.0
• Component/s: Compaction

Release Note

Enhance DateTieredCompaction to support HDFS storage policy within one class family.
# First you need enable DTCP.
To turn on Date Tiered Compaction (It is not recommended to turn on for the whole cluster because that will put meta table on it too and random get on meta table will be impacted):
hbase.hstore.compaction.compaction.policy=org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy
## Parameters for Date Tiered Compaction:
hbase.hstore.compaction.date.tiered.max.storefile.age.millis: Files with max-timestamp smaller than this will no longer be compacted.Default at Long.MAX_VALUE.
hbase.hstore.compaction.date.tiered.base.window.millis: base window size in milliseconds. Default at 6 hours.
hbase.hstore.compaction.date.tiered.windows.per.tier: number of windows per tier. Default at 4.
hbase.hstore.compaction.date.tiered.incoming.window.min: minimal number of files to compact in the incoming window. Set it to expected number of files in the window to avoid wasteful compaction. Default at 6.

# Then enable HDTCP(Heterogeneous Date Tiered Compaction) as follow example configurations:
hbase.hstore.compaction.date.tiered.storage.policy.enable=true
hbase.hstore.compaction.date.tiered.hot.window.age.millis=3600000
hbase.hstore.compaction.date.tiered.hot.window.storage.policy=ALL_SSD
hbase.hstore.compaction.date.tiered.warm.window.age.millis=20600000
hbase.hstore.compaction.date.tiered.warm.window.storage.policy=ONE_SSD
hbase.hstore.compaction.date.tiered.cold.window.storage.policy=HOT
## It is better to enable WAL and flushing HFile storage policy with HDTCP. You can tune follow settings as well:
hbase.wal.storage.policy=ALL_SSD
create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ALL_SSD'}}

# Disable HDTCP as follow:
hbase.hstore.compaction.date.tiered.storage.policy.enable=false

• Assignee: Mengqing Peng
• Reporter: Mengqing Peng
• Created: 30/Apr/20 04:48
• Updated: 18/Aug/20 01:18
• Resolved: 30/Jun/20 07:21

Description

Support DateTiredCompaction(HBASE-15181) for cold and hot data separation, support different storage policies for different time periods of data to get better performance, for example, we can configure the data of last 1 month in SSD, and 1 month ago data was in HDD.

design doc: 

https://docs.google.com/document/d/1fk_EWLNnxniwt3gDjUS_apQ3cPzn90AmvDT1wkirvKE/edit?usp=sharing

Issue Links

• links to: design doc
• links to: GitHub Pull Request #1730