Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17172

Optimize mob compaction with _del files

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • mob
    • None
    • Reviewed

    Description

      Today, when there is a _del file in mobdir, with major mob compaction, every mob file will be recompacted, this causes lots of IO and slow down major mob compaction (may take months to finish). This needs to be improved. A few ideas are:

      1) Do not compact all _del files into one, instead, compact them based on groups with startKey as the key. Then use firstKey/startKey to make each mob file to see if the _del file needs to be included for this partition.

      2). Based on the timerange of the _del file, compaction for files after that timerange does not need to include the _del file as these are newer files.

      Attachments

        1. HBASE-17172.master.001.patch
          40 kB
          Hua Xiang
        2. HBASE-17172.master.002.patch
          40 kB
          Hua Xiang
        3. HBASE-17172.master.003.patch
          40 kB
          Hua Xiang
        4. HBASE-17172-master-001.patch
          39 kB
          Hua Xiang

        Issue Links

          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. HBASE-16981 Expand Mob Compaction Partition policy from daily to weekly, monthly

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Review Board (master)

          Activity

            People

              huaxiang Hua Xiang
              huaxiang Hua Xiang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: