HBase
  1. HBase
  2. HBASE-3826

Minor compaction needs to check if still over compactionThreshold after compacting

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.1
    • Fix Version/s: 0.92.0
    • Component/s: regionserver
    • Labels:
    • Environment:

      hbase-0.90.1
      hbase-0.90.1-cdh3u0

    • Hadoop Flags:
      Reviewed
    • Tags:
      compaction

      Description

      I have a busy region, and there are 43 StoreFiles (>compactionThreshold=8) in this region.
      Now, I stopped the client and stopped putting new data into it. I expect these StoreFiles to be compacted later.

      But, almost one day later, these 43 StoreFiles are still there.
      (Note: in my hbase instance, I disabled the major compaction.)

      It seems the minor compaction does not be started continuiously to compact remaining storefiles.
      And I checked the code, it is true.


      After more test, a obvious issue/problem is, the complete of a minor compaction does not check if current storefiles need more minor compaction.

      I think this may be a bug or leak.

      Try this test:

      1. Put many data to a region, then there are 30 storefiles accumulated, because the backend compaction cannot catch up with the fast puts. (hbase.hstore.compactionThreshold=8, base.hstore.compaction.max=12)

      2. Then stop put.

      3. Then, these 30 storefiles are still there for a long time, (no automatic minor compaction)

      4. Submit a compaction on this region, then, only 12 files are compaction, now, we have 19 storefiles. The minor compaction stopped.

      I think, when a minor compaction complete, it should check if the number of storefiles still many, if so, another minor compaction should start continuiously.

      1. HBASE-3826.patch
        7 kB
        Nicolas Spiegelberg
      2. HBASE-3826_0.92.patch
        5 kB
        Nicolas Spiegelberg

        Activity

          People

          • Assignee:
            Nicolas Spiegelberg
            Reporter:
            Schubert Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development