Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14651

Default minimum compaction size is too high

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      hbase.hstore.compaction.min.size defines minimum selection size which is always eligible for minor compaction (no compaction ratio check is performed on such file selections). Default size is equals to memstore flush size (128MB). First of all, even this value is too high for some (many) deployments, especially for write intensive, because of a small sizes of a memstore flushes, and if user increases memstore flush size (they usually set it to at least 256MB), they have no idea how will it impact the overall compaction process efficiency. With 256MB of minimum size to compact, compactor most of the time skips necessary file ratio checks and this will result in increased read/write IO during compactions, because of the unbalanced selections where relatively large files can be mixed with a newly created small store files. I think we should set this default minimum to 64MB and not to link it to memstore flush size at all.

        Attachments

        1. HBASE-14651-v2.patch
          2 kB
          Vladimir Rodionov
        2. HBASE-14651-v1.patch
          1 kB
          Vladimir Rodionov
        3. files.png
          27 kB
          Michael Stack
        4. bytes.png
          24 kB
          Michael Stack

          Issue Links

            Activity

              People

              • Assignee:
                vrodionov Vladimir Rodionov
                Reporter:
                vrodionov Vladimir Rodionov
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: