Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This issue will track the implementation of how to handle the memory pressure for secondary region replicas. Since the replicas cannot flush by themselves, the region server might get blocked or cause extensive flushing for its primary regions. The design doc attached at HBASE-11183 contains two possible solutions that we can pursue. The first one is to not allow secondary region replicas to not flush by themselves, but instead of needed allow them to refresh their store files on demand (which possibly allows them to drop their memstore snapshots or memstores). The second approach is to allow the secondaries to flush to a temporary space.

      Both have pros and cons, but for simplicity and to not cause extra write amplification, we have implemented the first approach. More details can be found in the design doc, but we can also discuss other options here.

      1. hbase-12562_v3.patch
        23 kB
        Enis Soztutar
      2. hbase-12562_v2.patch
        22 kB
        Enis Soztutar
      3. hbase-12562_v2.patch
        22 kB
        Enis Soztutar
      4. hbase-12562_v2.patch
        22 kB
        Enis Soztutar
      5. hbase-12562_v1.patch
        23 kB
        Enis Soztutar

        Activity

        Hide
        ndimiduk Nick Dimiduk added a comment -

        Closing issues released in 1.1.0.

        Show
        ndimiduk Nick Dimiduk added a comment - Closing issues released in 1.1.0.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #6218 (See https://builds.apache.org/job/HBase-TRUNK/6218/)
        HBASE-12562 Handling memory pressure for secondary region replicas (enis: rev 4ac42a2f56b91ce864d1bcb04f1f9950e527aab1)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6218 (See https://builds.apache.org/job/HBase-TRUNK/6218/ ) HBASE-12562 Handling memory pressure for secondary region replicas (enis: rev 4ac42a2f56b91ce864d1bcb04f1f9950e527aab1) hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-1.1 #255 (See https://builds.apache.org/job/HBase-1.1/255/)
        HBASE-12562 Handling memory pressure for secondary region replicas (enis: rev 6e5e5d8ccef17c8d38f3284204bebd20f92525bd)

        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
        • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
        • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-1.1 #255 (See https://builds.apache.org/job/HBase-1.1/255/ ) HBASE-12562 Handling memory pressure for secondary region replicas (enis: rev 6e5e5d8ccef17c8d38f3284204bebd20f92525bd) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
        Hide
        enis Enis Soztutar added a comment -

        I've pushed this. Thanks for review.

        Show
        enis Enis Soztutar added a comment - I've pushed this. Thanks for review.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12703110/hbase-12562_v3.patch
        against master branch at commit 6d4a4a48fe90b889c39e3ed42448fc7597a46973.
        ATTACHMENT ID: 12703110

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703110/hbase-12562_v3.patch against master branch at commit 6d4a4a48fe90b889c39e3ed42448fc7597a46973. ATTACHMENT ID: 12703110 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13121//console This message is automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        v3 should fix findbugs warnings.

        Show
        enis Enis Soztutar added a comment - v3 should fix findbugs warnings.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12702961/hbase-12562_v2.patch
        against master branch at commit d58cce1694c22f8f895dceaaf2b2003f69d04ea4.
        ATTACHMENT ID: 12702961

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702961/hbase-12562_v2.patch against master branch at commit d58cce1694c22f8f895dceaaf2b2003f69d04ea4. ATTACHMENT ID: 12702961 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13112//console This message is automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        Something wrong with the branch pickup script it seems.

        Show
        enis Enis Soztutar added a comment - Something wrong with the branch pickup script it seems.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12702943/hbase-12562_v2.patch
        against 0.94 branch at commit d58cce1694c22f8f895dceaaf2b2003f69d04ea4.
        ATTACHMENT ID: 12702943

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13110//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702943/hbase-12562_v2.patch against 0.94 branch at commit d58cce1694c22f8f895dceaaf2b2003f69d04ea4. ATTACHMENT ID: 12702943 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13110//console This message is automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        with -p0 this time.

        Show
        enis Enis Soztutar added a comment - with -p0 this time.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12702941/hbase-12562_v2.patch
        against 0.94 branch at commit d58cce1694c22f8f895dceaaf2b2003f69d04ea4.
        ATTACHMENT ID: 12702941

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13109//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702941/hbase-12562_v2.patch against 0.94 branch at commit d58cce1694c22f8f895dceaaf2b2003f69d04ea4. ATTACHMENT ID: 12702941 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13109//console This message is automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        Attaching v2 patch addressing review comments.

        Show
        enis Enis Soztutar added a comment - Attaching v2 patch addressing review comments.
        Hide
        enis Enis Soztutar added a comment -

        Thanks Jeff for the review.

        You can break the loop after set canDrop to false

        done.

        2) Just to check acquiring lock on writestate and memstore are always in this order

        Good point. Moved the synchronized (this) block out.

        3) There maybe no need for the following condition

        Yes, it was there to skip doing refresh if the secondary is already in between replaying a start flush and commit flush. But removed that as a safe guard (we can end up with all regions in that state).

        4. Rename getBiggestMemstoreOfSecondaryRegion to getBiggestMemstoreOfRegionReplica may be better

        done

        Show
        enis Enis Soztutar added a comment - Thanks Jeff for the review. You can break the loop after set canDrop to false done. 2) Just to check acquiring lock on writestate and memstore are always in this order Good point. Moved the synchronized (this) block out. 3) There maybe no need for the following condition Yes, it was there to skip doing refresh if the secondary is already in between replaying a start flush and commit flush. But removed that as a safe guard (we can end up with all regions in that state). 4. Rename getBiggestMemstoreOfSecondaryRegion to getBiggestMemstoreOfRegionReplica may be better done
        Hide
        jeffreyz Jeffrey Zhong added a comment -

        +1. Looks good to me with some some minor comments:
        1)

        +          if (store.getSnapshotSize() > 0) {
        +            canDrop = false;
        +          }
        

        You can break the loop after set canDrop to false

        2) Just to check acquiring lock on writestate and memstore are always in this order

        3) There maybe no need for the following condition

        +        if (region.writestate.flushing
        

        4. Rename getBiggestMemstoreOfSecondaryRegion to getBiggestMemstoreOfRegionReplica may be better

        Show
        jeffreyz Jeffrey Zhong added a comment - +1. Looks good to me with some some minor comments: 1) + if (store.getSnapshotSize() > 0) { + canDrop = false ; + } You can break the loop after set canDrop to false 2) Just to check acquiring lock on writestate and memstore are always in this order 3) There maybe no need for the following condition + if (region.writestate.flushing 4. Rename getBiggestMemstoreOfSecondaryRegion to getBiggestMemstoreOfRegionReplica may be better
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12702261/hbase-12562_v1.patch
        against master branch at commit 883d6fd8e512b14c967d2f7acf78d2b1d40e40fe.
        ATTACHMENT ID: 12702261

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13083//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702261/hbase-12562_v1.patch against master branch at commit 883d6fd8e512b14c967d2f7acf78d2b1d40e40fe. ATTACHMENT ID: 12702261 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13083//console This message is automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        Here is a patch that implements a policy for freeing memstores by doing a "refresh store files" if necessary. Since secondary regions cannot flush by themselves, we use this extreme measure if needed.

        When there is global memory pressure, we have a chore to find out the best possible region to flush. This patch extends that to find the best possible primary region and best possible secondary region (biggest memstore), and then we compare their memstore sizes. We use a multiplier (default 4) for the comparison because we want to flush primary regions as much as possible, but not do refresh store files (because of semantics as detailed in the design doc).

        Show
        enis Enis Soztutar added a comment - Here is a patch that implements a policy for freeing memstores by doing a "refresh store files" if necessary. Since secondary regions cannot flush by themselves, we use this extreme measure if needed. When there is global memory pressure, we have a chore to find out the best possible region to flush. This patch extends that to find the best possible primary region and best possible secondary region (biggest memstore), and then we compare their memstore sizes. We use a multiplier (default 4) for the comparison because we want to flush primary regions as much as possible, but not do refresh store files (because of semantics as detailed in the design doc).

          People

          • Assignee:
            enis Enis Soztutar
            Reporter:
            enis Enis Soztutar
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development