Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-3826

Minor compaction needs to check if still over compactionThreshold after compacting

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.1
    • Fix Version/s: 0.92.0
    • Component/s: regionserver
    • Labels:
    • Environment:

      hbase-0.90.1
      hbase-0.90.1-cdh3u0

    • Hadoop Flags:
      Reviewed
    • Tags:
      compaction

      Description

      I have a busy region, and there are 43 StoreFiles (>compactionThreshold=8) in this region.
      Now, I stopped the client and stopped putting new data into it. I expect these StoreFiles to be compacted later.

      But, almost one day later, these 43 StoreFiles are still there.
      (Note: in my hbase instance, I disabled the major compaction.)

      It seems the minor compaction does not be started continuiously to compact remaining storefiles.
      And I checked the code, it is true.


      After more test, a obvious issue/problem is, the complete of a minor compaction does not check if current storefiles need more minor compaction.

      I think this may be a bug or leak.

      Try this test:

      1. Put many data to a region, then there are 30 storefiles accumulated, because the backend compaction cannot catch up with the fast puts. (hbase.hstore.compactionThreshold=8, base.hstore.compaction.max=12)

      2. Then stop put.

      3. Then, these 30 storefiles are still there for a long time, (no automatic minor compaction)

      4. Submit a compaction on this region, then, only 12 files are compaction, now, we have 19 storefiles. The minor compaction stopped.

      I think, when a minor compaction complete, it should check if the number of storefiles still many, if so, another minor compaction should start continuiously.

        Attachments

        1. HBASE-3826.patch
          7 kB
          Nicolas Spiegelberg
        2. HBASE-3826_0.92.patch
          5 kB
          Nicolas Spiegelberg

          Activity

            People

            • Assignee:
              nspiegelberg Nicolas Spiegelberg
              Reporter:
              schubertzhang Schubert Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: