HBase
  1. HBase
  2. HBASE-3826

Minor compaction needs to check if still over compactionThreshold after compacting

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.1
    • Fix Version/s: 0.92.0
    • Component/s: regionserver
    • Labels:
    • Environment:

      hbase-0.90.1
      hbase-0.90.1-cdh3u0

    • Hadoop Flags:
      Reviewed
    • Tags:
      compaction

      Description

      I have a busy region, and there are 43 StoreFiles (>compactionThreshold=8) in this region.
      Now, I stopped the client and stopped putting new data into it. I expect these StoreFiles to be compacted later.

      But, almost one day later, these 43 StoreFiles are still there.
      (Note: in my hbase instance, I disabled the major compaction.)

      It seems the minor compaction does not be started continuiously to compact remaining storefiles.
      And I checked the code, it is true.


      After more test, a obvious issue/problem is, the complete of a minor compaction does not check if current storefiles need more minor compaction.

      I think this may be a bug or leak.

      Try this test:

      1. Put many data to a region, then there are 30 storefiles accumulated, because the backend compaction cannot catch up with the fast puts. (hbase.hstore.compactionThreshold=8, base.hstore.compaction.max=12)

      2. Then stop put.

      3. Then, these 30 storefiles are still there for a long time, (no automatic minor compaction)

      4. Submit a compaction on this region, then, only 12 files are compaction, now, we have 19 storefiles. The minor compaction stopped.

      I think, when a minor compaction complete, it should check if the number of storefiles still many, if so, another minor compaction should start continuiously.

      1. HBASE-3826.patch
        7 kB
        Nicolas Spiegelberg
      2. HBASE-3826_0.92.patch
        5 kB
        Nicolas Spiegelberg

        Activity

        Hide
        Jean-Daniel Cryans added a comment -

        Editing the title and marking against 0.92

        Show
        Jean-Daniel Cryans added a comment - Editing the title and marking against 0.92
        Hide
        Nicolas Spiegelberg added a comment -

        I think recursive compactions are pretty dangerous. Although re-enqueuing after compaction is the optimal solution, Exceptions or other weird bugs in the compaction code would cause spin-waiting problems. I think the optimal solution is to have quick response for the blocking StoreFile case and asynchronous, rate-limited enqueuing for less severe cases.

        Show
        Nicolas Spiegelberg added a comment - I think recursive compactions are pretty dangerous. Although re-enqueuing after compaction is the optimal solution, Exceptions or other weird bugs in the compaction code would cause spin-waiting problems. I think the optimal solution is to have quick response for the blocking StoreFile case and asynchronous, rate-limited enqueuing for less severe cases.
        Hide
        Schubert Zhang added a comment -

        The patch seems re-queue compaction at reigion level.
        How about in store.

        Show
        Schubert Zhang added a comment - The patch seems re-queue compaction at reigion level. How about in store.
        Hide
        stack added a comment -

        @Nicolas How does this relate to 'HBASE-3796 Per-Store Entries in Compaction Queue'?

        Show
        stack added a comment - @Nicolas How does this relate to ' HBASE-3796 Per-Store Entries in Compaction Queue'?
        Hide
        stack added a comment -

        Marking patch available though there are some questions about the attached patch.

        Show
        stack added a comment - Marking patch available though there are some questions about the attached patch.
        Hide
        Nicolas Spiegelberg added a comment -

        Attached the 0.92 patch. Note that the recursive enqueue was added by HBASE-1476, so the only change necessary for this JIRA is to change the MajorCompactionChecker to a CompactionChecker.

        @Zhang : I assume since you want a Store-level diff, you want to use 0.92 & not 0.90 (the environment you reported). The old patch should work for 0.90.

        Show
        Nicolas Spiegelberg added a comment - Attached the 0.92 patch. Note that the recursive enqueue was added by HBASE-1476 , so the only change necessary for this JIRA is to change the MajorCompactionChecker to a CompactionChecker. @Zhang : I assume since you want a Store-level diff, you want to use 0.92 & not 0.90 (the environment you reported). The old patch should work for 0.90.
        Hide
        stack added a comment -

        Committed the 0.92 patch to TRUNK. Thanks Nicolas.

        Show
        stack added a comment - Committed the 0.92 patch to TRUNK. Thanks Nicolas.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #1930 (See https://builds.apache.org/hudson/job/HBase-TRUNK/1930/)

        Show
        Hudson added a comment - Integrated in HBase-TRUNK #1930 (See https://builds.apache.org/hudson/job/HBase-TRUNK/1930/ )

          People

          • Assignee:
            Nicolas Spiegelberg
            Reporter:
            Schubert Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development