Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      With the async wal approach (HBASE-11568), the edits are not persisted (to wal) in the secondary region replicas. However this means that we have to deal with secondary region replica failures.

      We can seek to re-replicate the edits from primary to the secondary when the secondary region is opened in another server but this would mean to setup a replication queue again, and holding on to the wals for longer.

      Instead, we can design it so that the edits form the secondaries are not persisted to wal, and if the secondary replica fails over, it will not start serving reads until it has guaranteed that it has all the past data.

      For guaranteeing that the secondary replica has all the edits before serving reads, we can use flush and region opening markers. Whenever a region open event is seen, it writes all the files at the time of opening to wal (HBASE-11512). In case of flush, the flushed file is written as well, and the secondary replica can do a ls for the store files and pick up all the files before the seqId of the flushed file. So, in this design, the secodary replica will wait until it sees and replays a flush or region open marker from wal from primary. and then start serving. For speeding up replica opening time, we can trigger a flush to the primary whenever the secondary replica opens as an optimization.

      1. hbase-11580_v2.patch
        142 kB
        Enis Soztutar
      2. hbase-11580_v3.patch
        142 kB
        Enis Soztutar
      3. hbase-11580-addendum.patch
        2 kB
        Enis Soztutar

        Issue Links

          Activity

          Hide
          Nick Dimiduk added a comment -

          Closing issues released in 1.1.0.

          Show
          Nick Dimiduk added a comment - Closing issues released in 1.1.0.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-1.1 #248 (See https://builds.apache.org/job/HBase-1.1/248/)
          HBASE-11580 Failover handling for secondary region replicas - ADDENDUM for javadoc fixes (enis: rev 07cb6d24130db15be303ca34e878bcf5d5411a42)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-1.1 #248 (See https://builds.apache.org/job/HBase-1.1/248/ ) HBASE-11580 Failover handling for secondary region replicas - ADDENDUM for javadoc fixes (enis: rev 07cb6d24130db15be303ca34e878bcf5d5411a42) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #6206 (See https://builds.apache.org/job/HBase-TRUNK/6206/)
          HBASE-11580 Failover handling for secondary region replicas - ADDENDUM for javadoc fixes (enis: rev d1ca560ff72bd97faa94fce025bf126702bbdd1b)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6206 (See https://builds.apache.org/job/HBase-TRUNK/6206/ ) HBASE-11580 Failover handling for secondary region replicas - ADDENDUM for javadoc fixes (enis: rev d1ca560ff72bd97faa94fce025bf126702bbdd1b) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Enis Soztutar added a comment -

          Committed this addendum. Sorry for the trouble.

          Show
          Enis Soztutar added a comment - Committed this addendum. Sorry for the trouble.
          Hide
          Enis Soztutar added a comment -

          Thanks Sean. I had searched the javadoc report before, but could not identify the warnings somehow. An oversight from my side. Will commit an addendum shortly.

          Show
          Enis Soztutar added a comment - Thanks Sean. I had searched the javadoc report before, but could not identify the warnings somehow. An oversight from my side. Will commit an addendum shortly.
          Hide
          Sean Busbey added a comment -

          this issue has introduced new javadoc warnings that are causing precommit checks for new patches to fail.

          [WARNING] Javadoc Warnings
          [WARNING] hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:1988: warning - Tag @see: can't find internalFlushcache(Collection, MonitoredTask) in org.apache.hadoop.hbase.regionserver.HRegion
          [WARNING] hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:1998: warning - Tag @see: can't find internalFlushcache(WAL, long, Collection, MonitoredTask) in org.apache.hadoop.hbase.regionserver.HRegion
          [WARNING] hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:2243: warning - @return tag has no arguments.
          [INFO]
          
          Show
          Sean Busbey added a comment - this issue has introduced new javadoc warnings that are causing precommit checks for new patches to fail. [WARNING] Javadoc Warnings [WARNING] hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:1988: warning - Tag @see: can't find internalFlushcache(Collection, MonitoredTask) in org.apache.hadoop.hbase.regionserver.HRegion [WARNING] hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:1998: warning - Tag @see: can't find internalFlushcache(WAL, long, Collection, MonitoredTask) in org.apache.hadoop.hbase.regionserver.HRegion [WARNING] hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:2243: warning - @return tag has no arguments. [INFO]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-1.1 #244 (See https://builds.apache.org/job/HBase-1.1/244/)
          HBASE-11580 Failover handling for secondary region replicas (enis: rev 9e7ad99593f6ce6f4f99e4ad67d271de51176f8f)

          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
          • hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RemoveColumnAction.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/FlushRegionCallable.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • hbase-protocol/src/main/protobuf/Admin.proto
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java
          • hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/util/RetryCounter.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
          • hbase-protocol/src/main/protobuf/WAL.proto
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaFailover.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-1.1 #244 (See https://builds.apache.org/job/HBase-1.1/244/ ) HBASE-11580 Failover handling for secondary region replicas (enis: rev 9e7ad99593f6ce6f4f99e4ad67d271de51176f8f) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RemoveColumnAction.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/FlushRegionCallable.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-protocol/src/main/protobuf/Admin.proto hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.java hbase-common/src/main/java/org/apache/hadoop/hbase/util/RetryCounter.java hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java hbase-protocol/src/main/protobuf/WAL.proto hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaFailover.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #6202 (See https://builds.apache.org/job/HBase-TRUNK/6202/)
          HBASE-11580 Failover handling for secondary region replicas (enis: rev 9899aab12b419144f7f8a8280bedbccc68ee7452)

          • hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RemoveColumnAction.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaFailover.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.java
          • hbase-protocol/src/main/protobuf/WAL.proto
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/util/RetryCounter.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/FlushRegionCallable.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • hbase-protocol/src/main/protobuf/Admin.proto
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6202 (See https://builds.apache.org/job/HBase-TRUNK/6202/ ) HBASE-11580 Failover handling for secondary region replicas (enis: rev 9899aab12b419144f7f8a8280bedbccc68ee7452) hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RemoveColumnAction.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicaFailover.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestRegionReplicaReplication.java hbase-protocol/src/main/protobuf/WAL.proto hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java hbase-common/src/main/java/org/apache/hadoop/hbase/util/RetryCounter.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/FlushRegionCallable.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java hbase-protocol/src/main/protobuf/Admin.proto hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
          Hide
          Enis Soztutar added a comment -

          I have committed this. Thanks Jeffrey for review.

          Show
          Enis Soztutar added a comment - I have committed this. Thanks Jeffrey for review.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702007/hbase-11580_v3.patch
          against master branch at commit 74e36f8ddd03cf94c17bdb30ecd81cc5dff4d063.
          ATTACHMENT ID: 12702007

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified tests.

          -1 Anti-pattern. The patch appears to have anti-pattern where BYTES_COMPARATOR was omitted:
          + getRegionInfo(), -1, new TreeMap<byte[], List<Path>>());.
          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

          +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702007/hbase-11580_v3.patch against master branch at commit 74e36f8ddd03cf94c17bdb30ecd81cc5dff4d063. ATTACHMENT ID: 12702007 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 23 new or modified tests. -1 Anti-pattern . The patch appears to have anti-pattern where BYTES_COMPARATOR was omitted: + getRegionInfo(), -1, new TreeMap<byte[], List<Path>>());. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13043//console This message is automatically generated.
          Hide
          Enis Soztutar added a comment -

          v3 addresses the unit test failure.

          Show
          Enis Soztutar added a comment - v3 addresses the unit test failure.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12701535/hbase-11580_v2.patch
          against master branch at commit bec2b0d320554b0af8c891fddc147a953f35765f.
          ATTACHMENT ID: 12701535

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified tests.

          -1 Anti-pattern. The patch appears to have anti-pattern where BYTES_COMPARATOR was omitted:
          + getRegionInfo(), -1, new TreeMap<byte[], List<Path>>());.
          +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

          +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestInterfaceAudienceAnnotations

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12701535/hbase-11580_v2.patch against master branch at commit bec2b0d320554b0af8c891fddc147a953f35765f. ATTACHMENT ID: 12701535 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 23 new or modified tests. -1 Anti-pattern . The patch appears to have anti-pattern where BYTES_COMPARATOR was omitted: + getRegionInfo(), -1, new TreeMap<byte[], List<Path>>());. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.TestInterfaceAudienceAnnotations Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13011//console This message is automatically generated.
          Hide
          Enis Soztutar added a comment -

          Thanks Jeffrey for the review. Attaching updated patch from RB.

          Show
          Enis Soztutar added a comment - Thanks Jeffrey for the review. Attaching updated patch from RB.
          Hide
          Jeffrey Zhong added a comment -

          I've reviewed the patch and left some comments on the review board. +1 assuming unit tests pass.

          For the "flush amplifications", it's more an optimization issue which can be addressed by secondary replica sends the seqId it firstly sees as part of flush request. Primary region can check if it's last flushed seqId is larger than the passed seqId from replica to decide if to perform a flush.

          Show
          Jeffrey Zhong added a comment - I've reviewed the patch and left some comments on the review board. +1 assuming unit tests pass. For the "flush amplifications", it's more an optimization issue which can be addressed by secondary replica sends the seqId it firstly sees as part of flush request. Primary region can check if it's last flushed seqId is larger than the passed seqId from replica to decide if to perform a flush.
          Hide
          Enis Soztutar added a comment -

          I have put up a patch for review for this. The RB description goes into some detail what the patch contains.

          Show
          Enis Soztutar added a comment - I have put up a patch for review for this. The RB description goes into some detail what the patch contains.
          Hide
          Enis Soztutar added a comment -

          Would this be on by default or would this be an option? It seems that in a failure situation we would hit a "flush amplification" problem which seems concerning

          I think we may want to turn this on by default, but we'll have to have some operational experience with the setting. You are right that there is a sync flush concern, but w/o making secondary replica edits durable in wal, there seems no other way other than waiting for a flush to start serving data from memstore.

          We can also add some jitter as well to cope with this.

          Show
          Enis Soztutar added a comment - Would this be on by default or would this be an option? It seems that in a failure situation we would hit a "flush amplification" problem which seems concerning I think we may want to turn this on by default, but we'll have to have some operational experience with the setting. You are right that there is a sync flush concern, but w/o making secondary replica edits durable in wal, there seems no other way other than waiting for a flush to start serving data from memstore. We can also add some jitter as well to cope with this.
          Hide
          Jonathan Hsieh added a comment -

          As an optimization so that the secondary will sooner start serving data, the region server opening a secondary region replica will make an RPC to the RS serving the primary for a flush. This will result either a flush, which will asynchronously be replayed to the secondary, or a no-op because memstores are empty. In the latter case, we should still drop a WAL marker though.

          Would this be on by default or would this be an option? It seems that in a failure situation we would hit a "flush amplification" problem which seems concerning.

          Let's say we have a 20 node cluster with each rs having 25 regions on them. If those regions have 2 secondaries, each hosts would have 50 secondaries. Let's say one rs goes down When these secondaries gets rebalanced, they'd get spread around a bunch likely landing on every or almost every node in the cluster. If each secondary region reopening triggers a flush request we'd end up with a lot of flush requests, all around the same time all across the cluster, no? This seems problematic. We could probably coalesce some of the flushes (each rs receiving secondaries could consolidate all into one flush) but if we had 5 machines go down wouldn't we still get 5x the flushes?

          Show
          Jonathan Hsieh added a comment - As an optimization so that the secondary will sooner start serving data, the region server opening a secondary region replica will make an RPC to the RS serving the primary for a flush. This will result either a flush, which will asynchronously be replayed to the secondary, or a no-op because memstores are empty. In the latter case, we should still drop a WAL marker though. Would this be on by default or would this be an option? It seems that in a failure situation we would hit a "flush amplification" problem which seems concerning. Let's say we have a 20 node cluster with each rs having 25 regions on them. If those regions have 2 secondaries, each hosts would have 50 secondaries. Let's say one rs goes down When these secondaries gets rebalanced, they'd get spread around a bunch likely landing on every or almost every node in the cluster. If each secondary region reopening triggers a flush request we'd end up with a lot of flush requests, all around the same time all across the cluster, no? This seems problematic. We could probably coalesce some of the flushes (each rs receiving secondaries could consolidate all into one flush) but if we had 5 machines go down wouldn't we still get 5x the flushes?
          Hide
          Enis Soztutar added a comment -

          Sorry I should have missed that one.

          The reason we need special handling in secondary region replicas is that the edits are replayed to secondaries, but the WAL is skipped in secondary replicas. This is because we do not want to cause additional WAL writes for secondaries.

          However, this implies that when the secondary region replica starts serving, it cannot make sure that it has all the memstore updates because some of it may have been missed after the last flush. So, what I am thinking is a mechanism with two parts:

          • Whenever a secondary is opened, it will reject read requests until it makes sure (on its own) that is has all the updates up to a seqId. The seqId has to be greater than the last flush seqId of the files since a previous open of the same region replica might have replayed some in-memory data, and we do not want the seqId served from a particular replica_id to go back. At this state, the secondary will watch for flush start and commit events from WAL replay. The secondary will ignore all replayed edits before seeing a flush start record. After seeing flush start, it will apply all edits to memstores. Once the secondary sees the corresponding flush commit, it can pick up the new flushed file, and start serving from all flushed files and its memstore data.
          • As an optimization so that the secondary will sooner start serving data, the region server opening a secondary region replica will make an RPC to the RS serving the primary for a flush. This will result either a flush, which will asynchronously be replayed to the secondary, or a no-op because memstores are empty. In the latter case, we should still drop a WAL marker though.
          Show
          Enis Soztutar added a comment - Sorry I should have missed that one. The reason we need special handling in secondary region replicas is that the edits are replayed to secondaries, but the WAL is skipped in secondary replicas. This is because we do not want to cause additional WAL writes for secondaries. However, this implies that when the secondary region replica starts serving, it cannot make sure that it has all the memstore updates because some of it may have been missed after the last flush. So, what I am thinking is a mechanism with two parts: Whenever a secondary is opened, it will reject read requests until it makes sure (on its own) that is has all the updates up to a seqId. The seqId has to be greater than the last flush seqId of the files since a previous open of the same region replica might have replayed some in-memory data, and we do not want the seqId served from a particular replica_id to go back. At this state, the secondary will watch for flush start and commit events from WAL replay. The secondary will ignore all replayed edits before seeing a flush start record. After seeing flush start, it will apply all edits to memstores. Once the secondary sees the corresponding flush commit, it can pick up the new flushed file, and start serving from all flushed files and its memstore data. As an optimization so that the secondary will sooner start serving data, the region server opening a secondary region replica will make an RPC to the RS serving the primary for a flush. This will result either a flush, which will asynchronously be replayed to the secondary, or a no-op because memstores are empty. In the latter case, we should still drop a WAL marker though.
          Hide
          Jonathan Hsieh added a comment -

          This wasn't answered in HBASE-11183

          enis: Whenever a secondary starts serving, it will trigger a flush from the primary region ..
          stack: How will this work?

          Do you have a sense on what the policy will be? can you fill in more details about how this would work?

          Show
          Jonathan Hsieh added a comment - This wasn't answered in HBASE-11183 enis: Whenever a secondary starts serving, it will trigger a flush from the primary region .. stack: How will this work? Do you have a sense on what the policy will be? can you fill in more details about how this would work?

            People

            • Assignee:
              Enis Soztutar
              Reporter:
              Enis Soztutar
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development