Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17172

Optimize mob compaction with _del files

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: mob
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Today, when there is a _del file in mobdir, with major mob compaction, every mob file will be recompacted, this causes lots of IO and slow down major mob compaction (may take months to finish). This needs to be improved. A few ideas are:

      1) Do not compact all _del files into one, instead, compact them based on groups with startKey as the key. Then use firstKey/startKey to make each mob file to see if the _del file needs to be included for this partition.

      2). Based on the timerange of the _del file, compaction for files after that timerange does not need to include the _del file as these are newer files.

        Attachments

        1. HBASE-17172.master.001.patch
          40 kB
          Hua Xiang
        2. HBASE-17172.master.002.patch
          40 kB
          Hua Xiang
        3. HBASE-17172.master.003.patch
          40 kB
          Hua Xiang
        4. HBASE-17172-master-001.patch
          39 kB
          Hua Xiang

          Issue Links

            Activity

              People

              • Assignee:
                huaxiang Hua Xiang
                Reporter:
                huaxiang Hua Xiang
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: