Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24601

Change default Hfile storage policy from HOT to NONE for HDFS



    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.5
    • 2.5.0, 3.0.0-alpha-2
    • HFile
    • None
    • Reviewed


      The current HBase hfile directory specifies the HOT storage strategy by default(`hbase.hstore.blockingStoreFiles` ), which is different from the default value of HDFS heterogeneous strategy now(should be empty)

      It should be set to NONE like `hbase.wal.storage.policy`'s default value on HBASE-20691, and the reason for modification is the same

      Otherwise, it could have a significant impact because most HDFS Managers doesn't know the default value has been changed to HOT by hbase, for example:

      HDFS manger want to set hfile's dir to ONE_SSD for speed up read, they will do:

      // Set Hbas Hfile dir to ONE_SSD for speed up read
      // The policy that expects all subdirectories to inherit the parent directory by default (empty by default)
      bin/hdfs storagepolicies -setStoragePolicy -path /home/hbase/data  -policy ONE_SSD
      bin/hdfs storagepolicies -setStoragePolicy -path /home/hbase/WALs  -policy ALL_SSD
      // If we do not change Hbase default setting, we will find this
      bin/hdfs storagepolicies -getStoragePolicy -path /home/hbase/data/hbase/meta/xxxx/info 
      return "HOT" instead of "ONE_SSD"

      And here is a pic to show the problem ↑:


      However, if Hfile uses the HOT strategy by default, then we will find that the actually stored directory has been set to HOT, then it will not inherit the ONE_SSD of the parent directory policy, and if we set NONE in hbase, this is problem can be avoided

       Unless someone sets other values manually, then we believe the operator knows what he is doing


        1. hfile-policy-problem.png
          33 kB

        Issue Links



              dark_num dark_num
              dark_num dark_num
              0 Vote for this issue
              8 Start watching this issue