Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Similar to writing flush events to WAL (HBASE-11511) and compaction events to WAL (HBASE-2231), we should write region open and close events to WAL.

      This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas.

      However, we may need this for regular inter cluster replication as well, see issues HBASE-10343 and HBASE-9465.

      A design doc for secondary replica replication can be found at HBASE-11183.

      1. hbase-11512_v3.patch
        124 kB
        Enis Soztutar
      2. hbase-11512_v2.patch
        136 kB
        Enis Soztutar
      3. hbase-11512_v1.patch
        130 kB
        Enis Soztutar

        Issue Links

          Activity

          Hide
          Enis Soztutar added a comment -

          Attaching a patch in the same line as HBASE-11511. Both region open and region close events are persisted to WAL.

          Show
          Enis Soztutar added a comment - Attaching a patch in the same line as HBASE-11511 . Both region open and region close events are persisted to WAL.
          Hide
          Enis Soztutar added a comment -

          Created RB on request: https://reviews.apache.org/r/23560/

          Show
          Enis Soztutar added a comment - Created RB on request: https://reviews.apache.org/r/23560/
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12655885/hbase-11512_v1.patch
          against trunk revision .
          ATTACHMENT ID: 12655885

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + new java.lang.String[]

          { "EventType", "TableName", "EncodedRegionName", "LogSequenceNumber", "Stores", "Server", }

          );
          + verify(log, times(1)).appendNoSync((HTableDescriptor)any(), (HRegionInfo)any(), (HLogKey)any(),

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestDistributedLogSplitting
          org.apache.hadoop.hbase.client.TestReplicaWithCluster

          -1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:286)
          at org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:260)
          at org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:177)
          at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testWorkerAbort(TestDistributedLogSplitting.java:956)

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655885/hbase-11512_v1.patch against trunk revision . ATTACHMENT ID: 12655885 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + new java.lang.String[] { "EventType", "TableName", "EncodedRegionName", "LogSequenceNumber", "Stores", "Server", } ); + verify(log, times(1)).appendNoSync((HTableDescriptor)any(), (HRegionInfo)any(), (HLogKey)any(), +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.client.TestReplicaWithCluster -1 core zombie tests . There are 2 zombie test(s): at org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:286) at org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:260) at org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:177) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testWorkerAbort(TestDistributedLogSplitting.java:956) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10095//console This message is automatically generated.
          Hide
          Enis Soztutar added a comment -

          v2 patch from RB.

          Show
          Enis Soztutar added a comment - v2 patch from RB.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12657428/hbase-11512_v2.patch
          against trunk revision .
          ATTACHMENT ID: 12657428

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657428/hbase-11512_v2.patch against trunk revision . ATTACHMENT ID: 12657428 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10164//console This message is automatically generated.
          Hide
          Jeffrey Zhong added a comment -

          I've reviewed the patch. Looks good to me(+1) except a minor thing I posted in RB. Thanks.

          Show
          Jeffrey Zhong added a comment - I've reviewed the patch. Looks good to me(+1) except a minor thing I posted in RB. Thanks.
          Hide
          Enis Soztutar added a comment -

          v3 patch from RB.

          Show
          Enis Soztutar added a comment - v3 patch from RB.
          Hide
          Jeffrey Zhong added a comment -

          v3 looks good to me(+1). Thanks.

          Show
          Jeffrey Zhong added a comment - v3 looks good to me(+1). Thanks.
          Hide
          Enis Soztutar added a comment -

          I've committed this to master and branch-1. Thanks for reviews.

          Show
          Enis Soztutar added a comment - I've committed this to master and branch-1. Thanks for reviews.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5413 (See https://builds.apache.org/job/HBase-TRUNK/5413/)
          HBASE-11512 Write region open/close events to WAL (enis: rev d44e7df5dc1e701fae4b611b9858dfd80499ee35)

          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • hbase-protocol/src/main/protobuf/WAL.proto
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5413 (See https://builds.apache.org/job/HBase-TRUNK/5413/ ) HBASE-11512 Write region open/close events to WAL (enis: rev d44e7df5dc1e701fae4b611b9858dfd80499ee35) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java hbase-protocol/src/main/protobuf/WAL.proto hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-1.0 #114 (See https://builds.apache.org/job/HBase-1.0/114/)
          HBASE-11512 Write region open/close events to WAL (enis: rev cbc1bc7d49c19255081eb3c6a5818dee81fce170)

          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • hbase-protocol/src/main/protobuf/WAL.proto
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java
          • hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-1.0 #114 (See https://builds.apache.org/job/HBase-1.0/114/ ) HBASE-11512 Write region open/close events to WAL (enis: rev cbc1bc7d49c19255081eb3c6a5818dee81fce170) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java hbase-protocol/src/main/protobuf/WAL.proto hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
          Hide
          Enis Soztutar added a comment -

          Closing this issue after 0.99.0 release.

          Show
          Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

            People

            • Assignee:
              Enis Soztutar
              Reporter:
              Enis Soztutar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development