HBase
  1. HBase
  2. HBASE-1947

If HBase starts/stops often in less than 24 hours, you end up with lots of store files

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.20.2, 0.90.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Since we don't compact on open and close of regions, all regions that have edits will do a flush when the cluster is stopped so that if you do it a couple of times in a row it's easy to get more than 10 store files and it won't be compacted until the next day. On open, we should check that number and compact if needed.

      1. HBASE-1947.patch
        3 kB
        Jean-Daniel Cryans

        Activity

        Jean-Daniel Cryans created issue -
        Hide
        Jean-Daniel Cryans added a comment -

        Simple patch that adds new methods to do the checks. I tried it on my machine and it compacted the 20 store files I had in ROOT right away.

        Show
        Jean-Daniel Cryans added a comment - Simple patch that adds new methods to do the checks. I tried it on my machine and it compacted the 20 store files I had in ROOT right away.
        Jean-Daniel Cryans made changes -
        Field Original Value New Value
        Attachment HBASE-1947.patch [ 12423721 ]
        Hide
        Jean-Daniel Cryans added a comment -

        Ryan reviewed this patch and wants me to clear the compaction message. I will commit the same patch but with:

        -        // Startup a compaction early if one is needed, if region has references.
        -        if (region.hasReferences()) {
        +        // Startup a compaction early if one is needed, if region has references
        +        // or it a store has too many store files
        +        if (region.hasReferences() || region.hasTooManyStoreFiles()) {
                   this.compactSplitThread.compactionRequested(region,
        -            "Region has references on open");
        +            region.hasReferences() ? "Region has references on open" :
        +                                     "Region has too many store files");
        
        Show
        Jean-Daniel Cryans added a comment - Ryan reviewed this patch and wants me to clear the compaction message. I will commit the same patch but with: - // Startup a compaction early if one is needed, if region has references. - if (region.hasReferences()) { + // Startup a compaction early if one is needed, if region has references + // or it a store has too many store files + if (region.hasReferences() || region.hasTooManyStoreFiles()) { this .compactSplitThread.compactionRequested(region, - "Region has references on open" ); + region.hasReferences() ? "Region has references on open" : + "Region has too many store files" );
        Hide
        Jean-Daniel Cryans added a comment -

        Committed to branch and trunk.

        Show
        Jean-Daniel Cryans added a comment - Committed to branch and trunk.
        Jean-Daniel Cryans made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Assignee Jean-Daniel Cryans [ jdcryans ]
        Resolution Fixed [ 1 ]
        stack made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jean-Daniel Cryans
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development