[HBASE-14061] Support CF-level Storage Policy

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: HFile, regionserver

Environment

hadoop-2.6.0

• Hadoop Flags: Reviewed

Release Note

After HBASE-14061 we support to set storage policy for HFile through "hbase.hstore.block.storage.policy" configuration, and we support CF-level setting to override the settings from configuration file. Currently supported storage policies include ALL_SSD/ONE_SSD/HOT/WARM/COLD, refer to http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html for more details

For example, to create a table with two families: "cf1" with "ALL_SSD" storage policy and "cf2" with "ONE_SSD", we could use below command in hbase shell:
create 'table',{NAME=>'f1',STORAGE_POLICY=>'ALL_SSD'},{NAME=>'f2',STORAGE_POLICY=>'ONE_SSD'}

We could also set the configuration in table attribute like all other configurations:
create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ONE_SSD'}}

• Assignee: Yu Li
• Reporter: Victor Xu
• Created: 12/Jul/15 06:38
• Updated: 01/Jul/22 20:21
• Resolved: 11/Jan/17 10:28

Description

After reading HBASE-12848 and HBASE-12934, I wrote a patch to implement cf-level storage policy.
My main purpose is to improve random-read performance for some really hot data, which usually locates in certain column family of a big table.

Usage:
$ hbase shell
> alter 'TABLE_NAME', METADATA =>

{'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}

> alter 'TABLE_NAME', {NAME=>'CF_NAME', METADATA => {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}}

HDFS's setStoragePolicy can only take effect when new hfile is created in a configured directory, so I had to make sub directories(for each cf) in region's .tmp directory and set storage policy for them.

Besides, I had to upgrade hadoop version to 2.6.0 because dfs.getStoragePolicy cannot be easily written in reflection, and I needed this api to finish my unit test.

Issue Links

• Is contained by: HBASE-15161 Umbrella: Miscellaneous improvements from production usage
• is related to: HBASE-17474 Reduce frequency of NoSuchMethodException when calling setStoragePolicy()
• is related to: HBASE-19858 Backport HBASE-14061 (Support CF-level Storage Policy) to branch-1
• relates to: HBASE-20691 Storage policy should allow deferring to HDFS
• relates to: HADOOP-12161 Add getStoragePolicy API to the FileSystem interface
• relates to: HDFS-8361 Choose SSD over DISK in block placement
• relates to: HDFS-9666 Enable hdfs-client to read even remote SSD/RAM prior to local disk replica to improve random read