Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-16981

Expand Mob Compaction Partition policy from daily to weekly, monthly

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: mob
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Mob compaction partition policy can be set by
      hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
       
      or

      hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY => 'monthly'}

      Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and "monthly", the default is "daily".

      When it is "weekly" policy, the mob compaction will try to compact files within one calendar week into one for a specific partition, similar for "daily" and "monthly".

      With "weekly" policy, one mob file normally is compacted twice during its lifetime (that is first on daily basis and then all such daily based compacted files belonging to a week at the weekly interval), for one region, there normally are 52 files for one year. With "Monthly" policy, one mob file normally is compacted 3 times during its lifetime (First daily and then weekly followed by monthly at end of every month) and normally there are 12 files for one year.
      Show
      Mob compaction partition policy can be set by hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY => 'weekly'}   or hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY => 'monthly'} Available MOB_COMPACT_PARTITION_POLICY options are "daily", "weekly" and "monthly", the default is "daily". When it is "weekly" policy, the mob compaction will try to compact files within one calendar week into one for a specific partition, similar for "daily" and "monthly". With "weekly" policy, one mob file normally is compacted twice during its lifetime (that is first on daily basis and then all such daily based compacted files belonging to a week at the weekly interval), for one region, there normally are 52 files for one year. With "Monthly" policy, one mob file normally is compacted 3 times during its lifetime (First daily and then weekly followed by monthly at end of every month) and normally there are 12 files for one year.

      Description

      Today the mob region holds all mob files for all regions. With daily partition mob compaction policy, after major mob compaction, there is still one file per region daily. Given there is 365 days in one year, at least 365 files per region. Since HDFS has limitation for number of files under one folder, this is not going to scale if there are lots of regions. To reduce mob file number, we want to introduce other partition policies such as weekly, monthly to compact mob files within one week or month into one file. This jira is create to track this effort.

        Attachments

        1. Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
          80 kB
          Hua Xiang
        2. HBASE-16981.master.008.patch
          60 kB
          Hua Xiang
        3. HBASE-16981.master.007.patch
          58 kB
          Hua Xiang
        4. HBASE-16981.master.006.patch
          59 kB
          Hua Xiang
        5. HBASE-16981.master.005.patch
          58 kB
          Hua Xiang
        6. HBASE-16981.master.004.patch
          59 kB
          Hua Xiang
        7. HBASE-16981.master.003.patch
          59 kB
          Hua Xiang
        8. HBASE-16981.master.002.patch
          45 kB
          Hua Xiang
        9. HBASE-16981.master.001.patch
          45 kB
          Hua Xiang

          Issue Links

            Activity

              People

              • Assignee:
                huaxiang Hua Xiang
                Reporter:
                huaxiang Hua Xiang
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: