HBase
  1. HBase
  2. HBASE-5867

Improve Compaction Throttle Default

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None

      Description

      We recently had a production issue where our compactions fell behind because our compaction throttle was improperly tuned and accidentally upgraded all compactions to the large pool. The default from HBASE-3877 makes 1 bad assumption: the default number of flushed files in a compaction. Currently the algorithm is:

      throttleSize ~= flushSize * 2

      This assumes that the basic compaction utilizes 3 files and that all 3 files are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the values were not very compressible. Both conditions should be taken into consideration. As a default, it is less damaging for the large thread to be slightly higher than it needs to be versus having everything accidentally promoted.

      1. ASF.LICENSE.NOT.GRANTED--D2943.1.patch
        4 kB
        Phabricator
      2. HBASE-5867-trunk.patch
        8 kB
        Nicolas Spiegelberg

        Issue Links

          Activity

          Hide
          Nicolas Spiegelberg added a comment -

          The most common type of compaction is compacting only flushed files. Assuming that there is no compression (the default), then the common compaction size should be:

          minFiles * flushSize
          

          The current idea is to support this operation and supporting a compaction with 1 previously-compacted file. Assuming no overlap, this size would be: (minFiles-1) * flushSize + minFiles * flushSize ==>

          2 * minFiles * flushSize - ε
          
          Show
          Nicolas Spiegelberg added a comment - The most common type of compaction is compacting only flushed files. Assuming that there is no compression (the default), then the common compaction size should be: minFiles * flushSize The current idea is to support this operation and supporting a compaction with 1 previously-compacted file. Assuming no overlap, this size would be: (minFiles-1) * flushSize + minFiles * flushSize ==> 2 * minFiles * flushSize - ε
          Hide
          Phabricator added a comment -

          nspiegelberg requested code review of "[jira] HBASE-5867 [89-fb] Improve Compaction Throttle Default".
          Reviewers: JIRA, Kannan, Liyin

          We recently had a production issue where our compactions fell
          behind because our compaction throttle was improperly tuned and
          accidentally upgraded all compactions to the large pool. The default
          from HBASE-3877 makes 1 bad assumption: the default number of flushed
          files in a compaction. MinFilesToCompact should be taken into
          consideration. As a default, it is less damaging for the large thread to
          be slightly higher than it needs to be and only get timed-majors versus
          having everything accidentally promoted.

          TEST PLAN

          • mvn test

          REVISION DETAIL
          https://reviews.facebook.net/D2943

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/6717/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - nspiegelberg requested code review of " [jira] HBASE-5867 [89-fb] Improve Compaction Throttle Default". Reviewers: JIRA, Kannan, Liyin We recently had a production issue where our compactions fell behind because our compaction throttle was improperly tuned and accidentally upgraded all compactions to the large pool. The default from HBASE-3877 makes 1 bad assumption: the default number of flushed files in a compaction. MinFilesToCompact should be taken into consideration. As a default, it is less damaging for the large thread to be slightly higher than it needs to be and only get timed-majors versus having everything accidentally promoted. TEST PLAN mvn test REVISION DETAIL https://reviews.facebook.net/D2943 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/Store.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6717/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Phabricator added a comment -

          Kannan has accepted the revision "[jira] HBASE-5867 [89-fb] Improve Compaction Throttle Default".

          looks good!

          REVISION DETAIL
          https://reviews.facebook.net/D2943

          BRANCH
          (no branch)

          Show
          Phabricator added a comment - Kannan has accepted the revision " [jira] HBASE-5867 [89-fb] Improve Compaction Throttle Default". looks good! REVISION DETAIL https://reviews.facebook.net/D2943 BRANCH (no branch)
          Hide
          Phabricator added a comment -

          stack has commented on the revision "[jira] HBASE-5867 [89-fb] Improve Compaction Throttle Default".

          Should we pull this into trunk Nicolas? Would it serve as good default for out-of-the-box hbase?

          REVISION DETAIL
          https://reviews.facebook.net/D2943

          BRANCH
          (no branch)

          Show
          Phabricator added a comment - stack has commented on the revision " [jira] HBASE-5867 [89-fb] Improve Compaction Throttle Default". Should we pull this into trunk Nicolas? Would it serve as good default for out-of-the-box hbase? REVISION DETAIL https://reviews.facebook.net/D2943 BRANCH (no branch)
          Hide
          Nicolas Spiegelberg added a comment -

          @stack: this should be a much better out-of-the-box than we currently have. If anything, we could make the value higher (4 or 8 instead of 2). However, this should be a substantially-better default than the current.

          Show
          Nicolas Spiegelberg added a comment - @stack: this should be a much better out-of-the-box than we currently have. If anything, we could make the value higher (4 or 8 instead of 2). However, this should be a substantially-better default than the current.
          Hide
          Nicolas Spiegelberg added a comment -

          straightforward port from 89fb

          Show
          Nicolas Spiegelberg added a comment - straightforward port from 89fb
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525641/HBASE-5867-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525641/HBASE-5867-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//console This message is automatically generated.
          Hide
          Phabricator added a comment -

          mbautin has committed the revision "[jira] HBASE-5867 [89-fb] Improve Compaction Throttle Default".

          REVISION DETAIL
          https://reviews.facebook.net/D2943

          COMMIT
          https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1334388

          Show
          Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-5867 [89-fb] Improve Compaction Throttle Default". REVISION DETAIL https://reviews.facebook.net/D2943 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1334388
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #5 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/5/)
          HBASE-5867 Improve Compaction Throttle Default

          Summary:
          We recently had a production issue where our compactions fell
          behind because our compaction throttle was improperly tuned and
          accidentally upgraded all compactions to the large pool. The default
          from HBASE-3877 makes 1 bad assumption: the default number of flushed
          files in a compaction. MinFilesToCompact should be taken into
          consideration. As a default, it is less damaging for the large thread
          to be slightly higher than it needs to be and only get timed-majors
          versus having everything accidentally promoted.

          Test Plan: - mvn test

          Reviewers: JIRA, Kannan, Liyin
          Reviewed By: Kannan
          CC: stack

          Differential Revision: https://reviews.facebook.net/D2943 (Revision 1338809)

          Result = FAILURE
          nspiegelberg :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #5 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/5/ ) HBASE-5867 Improve Compaction Throttle Default Summary: We recently had a production issue where our compactions fell behind because our compaction throttle was improperly tuned and accidentally upgraded all compactions to the large pool. The default from HBASE-3877 makes 1 bad assumption: the default number of flushed files in a compaction. MinFilesToCompact should be taken into consideration. As a default, it is less damaging for the large thread to be slightly higher than it needs to be and only get timed-majors versus having everything accidentally promoted. Test Plan: - mvn test Reviewers: JIRA, Kannan, Liyin Reviewed By: Kannan CC: stack Differential Revision: https://reviews.facebook.net/D2943 (Revision 1338809) Result = FAILURE nspiegelberg : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #554 (See https://builds.apache.org/job/HBase-0.94/554/)
          HBASE-7040 Port HBASE-5867 Improve Compaction Throttle Default to 0.94 (Sergey Shelukhin) (Revision 1402215)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #554 (See https://builds.apache.org/job/HBase-0.94/554/ ) HBASE-7040 Port HBASE-5867 Improve Compaction Throttle Default to 0.94 (Sergey Shelukhin) (Revision 1402215) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/)
          HBASE-7040 Port HBASE-5867 Improve Compaction Throttle Default to 0.94 (Sergey Shelukhin) (Revision 1402215)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/ ) HBASE-7040 Port HBASE-5867 Improve Compaction Throttle Default to 0.94 (Sergey Shelukhin) (Revision 1402215) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java

            People

            • Assignee:
              Nicolas Spiegelberg
              Reporter:
              Nicolas Spiegelberg
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development