XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • HFile, regionserver
    • None
    • hadoop-2.6.0

    • Reviewed
    • Hide
      After HBASE-14061 we support to set storage policy for HFile through "hbase.hstore.block.storage.policy" configuration, and we support CF-level setting to override the settings from configuration file. Currently supported storage policies include ALL_SSD/ONE_SSD/HOT/WARM/COLD, refer to http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html for more details

      For example, to create a table with two families: "cf1" with "ALL_SSD" storage policy and "cf2" with "ONE_SSD", we could use below command in hbase shell:
      create 'table',{NAME=>'f1',STORAGE_POLICY=>'ALL_SSD'},{NAME=>'f2',STORAGE_POLICY=>'ONE_SSD'}

      We could also set the configuration in table attribute like all other configurations:
      create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ONE_SSD'}}
      Show
      After HBASE-14061 we support to set storage policy for HFile through "hbase.hstore.block.storage.policy" configuration, and we support CF-level setting to override the settings from configuration file. Currently supported storage policies include ALL_SSD/ONE_SSD/HOT/WARM/COLD, refer to http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html for more details For example, to create a table with two families: "cf1" with "ALL_SSD" storage policy and "cf2" with "ONE_SSD", we could use below command in hbase shell: create 'table',{NAME=>'f1',STORAGE_POLICY=>'ALL_SSD'},{NAME=>'f2',STORAGE_POLICY=>'ONE_SSD'} We could also set the configuration in table attribute like all other configurations: create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ONE_SSD'}}

    Description

      After reading HBASE-12848 and HBASE-12934, I wrote a patch to implement cf-level storage policy.
      My main purpose is to improve random-read performance for some really hot data, which usually locates in certain column family of a big table.

      Usage:
      $ hbase shell
      > alter 'TABLE_NAME', METADATA =>

      {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}

      > alter 'TABLE_NAME', {NAME=>'CF_NAME', METADATA => {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}}

      HDFS's setStoragePolicy can only take effect when new hfile is created in a configured directory, so I had to make sub directories(for each cf) in region's .tmp directory and set storage policy for them.

      Besides, I had to upgrade hadoop version to 2.6.0 because dfs.getStoragePolicy cannot be easily written in reflection, and I needed this api to finish my unit test.

      Attachments

        1. HBASE-14061-master-v1.patch
          18 kB
          Victor Xu
        2. HBASE-14061.v2.patch
          18 kB
          Yu Li
        3. HBASE-14061.v3.patch
          23 kB
          Yu Li
        4. HBASE-14061.v4.patch
          23 kB
          Yu Li
        5. HBASE-14061.addendum.patch
          19 kB
          Yu Li
        6. HBASE-14061.addendum.patch
          19 kB
          Yu Li
        7. HBASE-14061.addendum2.patch
          0.8 kB
          Yu Li
        8. HBASE-14061.addendum2.patch
          0.8 kB
          Yu Li

        Issue Links

          Activity

            People

              liyu Yu Li
              victorunique Victor Xu
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: