Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24289

Heterogeneous Storage for Date Tiered Compaction

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha-1, 2.4.0
    • Component/s: Compaction
    • Labels:
      None
    • Release Note:
      Hide
      Enhance DateTieredCompaction to support HDFS storage policy within one class family.
      # First you need enable DTCP.
      To turn on Date Tiered Compaction (It is not recommended to turn on for the whole cluster because that will put meta table on it too and random get on meta table will be impacted):
      hbase.hstore.compaction.compaction.policy=org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy
      ## Parameters for Date Tiered Compaction:
      hbase.hstore.compaction.date.tiered.max.storefile.age.millis: Files with max-timestamp smaller than this will no longer be compacted.Default at Long.MAX_VALUE.
      hbase.hstore.compaction.date.tiered.base.window.millis: base window size in milliseconds. Default at 6 hours.
      hbase.hstore.compaction.date.tiered.windows.per.tier: number of windows per tier. Default at 4.
      hbase.hstore.compaction.date.tiered.incoming.window.min: minimal number of files to compact in the incoming window. Set it to expected number of files in the window to avoid wasteful compaction. Default at 6.

      # Then enable HDTCP(Heterogeneous Date Tiered Compaction) as follow example configurations:
      hbase.hstore.compaction.date.tiered.storage.policy.enable=true
      hbase.hstore.compaction.date.tiered.hot.window.age.millis=3600000
      hbase.hstore.compaction.date.tiered.hot.window.storage.policy=ALL_SSD
      hbase.hstore.compaction.date.tiered.warm.window.age.millis=20600000
      hbase.hstore.compaction.date.tiered.warm.window.storage.policy=ONE_SSD
      hbase.hstore.compaction.date.tiered.cold.window.storage.policy=HOT
      ## It is better to enable WAL and flushing HFile storage policy with HDTCP. You can tune follow settings as well:
      hbase.wal.storage.policy=ALL_SSD
      create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ALL_SSD'}}

      # Disable HDTCP as follow:
      hbase.hstore.compaction.date.tiered.storage.policy.enable=false
      Show
      Enhance DateTieredCompaction to support HDFS storage policy within one class family. # First you need enable DTCP. To turn on Date Tiered Compaction (It is not recommended to turn on for the whole cluster because that will put meta table on it too and random get on meta table will be impacted): hbase.hstore.compaction.compaction.policy=org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy ## Parameters for Date Tiered Compaction: hbase.hstore.compaction.date.tiered.max.storefile.age.millis: Files with max-timestamp smaller than this will no longer be compacted.Default at Long.MAX_VALUE. hbase.hstore.compaction.date.tiered.base.window.millis: base window size in milliseconds. Default at 6 hours. hbase.hstore.compaction.date.tiered.windows.per.tier: number of windows per tier. Default at 4. hbase.hstore.compaction.date.tiered.incoming.window.min: minimal number of files to compact in the incoming window. Set it to expected number of files in the window to avoid wasteful compaction. Default at 6. # Then enable HDTCP(Heterogeneous Date Tiered Compaction) as follow example configurations: hbase.hstore.compaction.date.tiered.storage.policy.enable=true hbase.hstore.compaction.date.tiered.hot.window.age.millis=3600000 hbase.hstore.compaction.date.tiered.hot.window.storage.policy=ALL_SSD hbase.hstore.compaction.date.tiered.warm.window.age.millis=20600000 hbase.hstore.compaction.date.tiered.warm.window.storage.policy=ONE_SSD hbase.hstore.compaction.date.tiered.cold.window.storage.policy=HOT ## It is better to enable WAL and flushing HFile storage policy with HDTCP. You can tune follow settings as well: hbase.wal.storage.policy=ALL_SSD create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ALL_SSD'}} # Disable HDTCP as follow: hbase.hstore.compaction.date.tiered.storage.policy.enable=false

      Description

      Support DateTiredCompaction(HBASE-15181) for cold and hot data separation, support different storage policies for different time periods of data to get better performance, for example, we can configure the data of last 1 month in SSD, and 1 month ago data was in HDD.

      design doc: 

      https://docs.google.com/document/d/1fk_EWLNnxniwt3gDjUS_apQ3cPzn90AmvDT1wkirvKE/edit?usp=sharing

        Attachments

          Activity

            People

            • Assignee:
              pengmq1 Mengqing Peng
              Reporter:
              pengmq1 Mengqing Peng
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: