Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-26229

Limit count and size of L0 files compaction in StripeCompactionPolicy

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.5.0, 3.0.0-alpha-2, 2.4.7
    • Compaction
    • None

    Description

      When selecting L0 files in the stripe store file manager to compact, they all will be selected. This is the key problem. No file count control and no compaction size control for L0 files compactions now. If the compaction size is large, e.g. some TBs, then the L0 compaction will need a lot of time to complete. 

      Since L0 files not only contains the recently flushed files, bulk loaded files will also be put into L0. And what's more, when opening a daughter region, if the parent stripes can not be rebuild in the daughter, all the files will be put to L0.

      So when there are large enough files in L0, there will exists a quite long compaction for all the L0 files. If the compaction speed less than the file flush speed to L0, larger compactions afterwards. This is a big problem especially in bulkloading files. 

       

      Attachments

        1. after.png
          663 kB
          Xiaolin Ha
        2. before.png
          160 kB
          Xiaolin Ha

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Xiaolin Ha Xiaolin Ha
            Xiaolin Ha Xiaolin Ha
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment