Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Disabled Distributed Log Replay (DLR) as default WAL/hlog splitting mechanism. 1.0.x releases will default to Distributed Log Splitting (DLS) system which is the default for 0.98, 0.96 and 0.94 releases.

      DLR improves MTTR (the more regions a server is carrying, the better the improvement. Also, we can start taking writes during WAL/hlog recovery), but rolling upgrade to HBase-1.0.x from earlier releases of 0.98 branch (0.98.0,..,0.98.3) can result in data loss. If you are not doing rolling upgrade, or you are rolling upgrading from a 0.98 greater than or equal to 0.98.4, you can safely turn on distributed log replay by default. Regular upgrade (as opposed to rolling upgrade) is not affected. See HBASE-11094 for details.

      See HBASE-7006 to learn more about DLR. You can enable this feature by setting hbase.master.distributed.log.replay to true in your hbase-site.xml file.
      Show
      Disabled Distributed Log Replay (DLR) as default WAL/hlog splitting mechanism. 1.0.x releases will default to Distributed Log Splitting (DLS) system which is the default for 0.98, 0.96 and 0.94 releases. DLR improves MTTR (the more regions a server is carrying, the better the improvement. Also, we can start taking writes during WAL/hlog recovery), but rolling upgrade to HBase-1.0.x from earlier releases of 0.98 branch (0.98.0,..,0.98.3) can result in data loss. If you are not doing rolling upgrade, or you are rolling upgrading from a 0.98 greater than or equal to 0.98.4, you can safely turn on distributed log replay by default. Regular upgrade (as opposed to rolling upgrade) is not affected. See HBASE-11094 for details. See HBASE-7006 to learn more about DLR. You can enable this feature by setting hbase.master.distributed.log.replay to true in your hbase-site.xml file.

      Description

      Distributed log replay is an awesome feature, but due of HBASE-11094, the rolling upgrade story from 0.98 is hard to explain / enforce.

      The fix for HBASE-11094 only went into 0.98.4, meaning rolling upgrades from 0.98.4- might lose data during the upgrade.

      I feel no matter how much documentation / warning we do, we cannot prevent users from doing rolling upgrades from 0.98.4- to 1.0. And we do not want to inconvenience the user by requiring a two step rolling upgrade.

      Thus I think we should disable dist log replay for 1.0, and re-enable it again for 1.1 (if rolling upgrade from 0.98 is not supported).

      ie. undo: HBASE-10888

      1. HBASE-12567.patch
        1 kB
        Enis Soztutar
      2. HBASE-12567.patch
        1 kB
        Jeffrey Zhong

        Issue Links

          Activity

          Hide
          busbey Sean Busbey added a comment -

          +1, though I'd rather see the non-rolling-upgrade barrier move to 2.0.

          Show
          busbey Sean Busbey added a comment - +1, though I'd rather see the non-rolling-upgrade barrier move to 2.0.
          Hide
          jeffreyz Jeffrey Zhong added a comment -

          Submit for QA run. Thanks.

          Show
          jeffreyz Jeffrey Zhong added a comment - Submit for QA run. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12683925/HBASE-12567.patch
          against master branch at commit 57b38339db4683aab01cc92d9b0acac367b9ff36.
          ATTACHMENT ID: 12683925

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestHCM
          org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
          org.apache.hadoop.hbase.client.TestFromClientSide

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/checkstyle-aggregate.html

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12683925/HBASE-12567.patch against master branch at commit 57b38339db4683aab01cc92d9b0acac367b9ff36. ATTACHMENT ID: 12683925 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor org.apache.hadoop.hbase.client.TestFromClientSide Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//console This message is automatically generated.
          Hide
          stack stack added a comment -

          TestHCM is a known issue. The other fails should be fixed just now.

          Show
          stack stack added a comment - TestHCM is a known issue. The other fails should be fixed just now.
          Hide
          lhofhansl Lars Hofhansl added a comment -

          Hmm... It seems OK to me to make an exception and make it clear. 0.98.4. Distributed Log Replay is too cool to have to off by default.
          I forget now, is it on by default in 0.98 (I know, I know, I could just check)?

          Show
          lhofhansl Lars Hofhansl added a comment - Hmm... It seems OK to me to make an exception and make it clear. 0.98.4. Distributed Log Replay is too cool to have to off by default. I forget now, is it on by default in 0.98 (I know, I know, I could just check)?
          Hide
          enis Enis Soztutar added a comment -

          We have never turned it on for 0.98. 0.99.0 and 0.99.1 shipped with it being on, but they are non-production releases. HBASE-10888 is the first attempt to turn it on by default AFAIK.

          Show
          enis Enis Soztutar added a comment - We have never turned it on for 0.98. 0.99.0 and 0.99.1 shipped with it being on, but they are non-production releases. HBASE-10888 is the first attempt to turn it on by default AFAIK.
          Hide
          enis Enis Soztutar added a comment -

          Retry hadoopqa.

          Show
          enis Enis Soztutar added a comment - Retry hadoopqa.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12684344/HBASE-12567.patch
          against master branch at commit 5985b485943e4664b83fad2f2cfd089e6031b062.
          ATTACHMENT ID: 12684344

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 182 warning messages.

          +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/checkstyle-aggregate.html

          Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/patchJavadocWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12684344/HBASE-12567.patch against master branch at commit 5985b485943e4664b83fad2f2cfd089e6031b062. ATTACHMENT ID: 12684344 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 182 warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//console This message is automatically generated.
          Hide
          lhofhansl Lars Hofhansl added a comment -

          Thanks Enis Soztutar. Agreed then, let's have it off for 1.0.

          Show
          lhofhansl Lars Hofhansl added a comment - Thanks Enis Soztutar . Agreed then, let's have it off for 1.0.
          Hide
          stack stack added a comment -

          Lets write fat note in upgrade section that if you are on 0.98.4+ or clean setup, that you SHOULD enable it.

          Show
          stack stack added a comment - Lets write fat note in upgrade section that if you are on 0.98.4+ or clean setup, that you SHOULD enable it.
          Hide
          enis Enis Soztutar added a comment -

          Pushed to branch-1. Thanks Jeffrey.

          Show
          enis Enis Soztutar added a comment - Pushed to branch-1. Thanks Jeffrey.
          Hide
          stack stack added a comment -

          Added note to the 1.0 upgrade section on DLR.

          Show
          stack stack added a comment - Added note to the 1.0 upgrade section on DLR.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-1.0 #525 (See https://builds.apache.org/job/HBase-1.0/525/)
          HBASE-12577 Disable distributed log replay by default (Jeffrey Zhong) (enis: rev 5610732173dcbbe9bc2c82170780f096385274cd)

          • hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-1.0 #525 (See https://builds.apache.org/job/HBase-1.0/525/ ) HBASE-12577 Disable distributed log replay by default (Jeffrey Zhong) (enis: rev 5610732173dcbbe9bc2c82170780f096385274cd) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          Hide
          enis Enis Soztutar added a comment -

          Thanks Stack.

          Show
          enis Enis Soztutar added a comment - Thanks Stack.
          Hide
          stack stack added a comment -

          See HBASE-12746. More needed disabling DLR

          Show
          stack stack added a comment - See HBASE-12746 . More needed disabling DLR
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-1.0 #614 (See https://builds.apache.org/job/HBase-1.0/614/)
          HBASE-12746 [1.0.0RC0] Distributed Log Replay is on (HBASE-12577 was insufficient) (Jeffrey Zhong) (stack: rev 2df74fbd4a85de1e5325ba6ad8595f2081238e29)

          • hbase-common/src/main/resources/hbase-default.xml
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-1.0 #614 (See https://builds.apache.org/job/HBase-1.0/614/ ) HBASE-12746 [1.0.0RC0] Distributed Log Replay is on ( HBASE-12577 was insufficient) (Jeffrey Zhong) (stack: rev 2df74fbd4a85de1e5325ba6ad8595f2081238e29) hbase-common/src/main/resources/hbase-default.xml hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-1.0 #629 (See https://builds.apache.org/job/HBase-1.0/629/)
          Revert "HBASE-12746 [1.0.0RC0] Distributed Log Replay is on (HBASE-12577 was insufficient) (Jeffrey Zhong)" (stack: rev d5f63f122131a45b24b5e3839d4b59a578652b2d)

          • hbase-common/src/main/resources/hbase-default.xml
          • hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
            HBASE-12746 [1.0.0RC0] Distributed Log Replay is on (HBASE-12577 was insufficient) (Jeffrey Zhong) (stack: rev a32f485fecbcf7e1aa7159ac2fa9851c1bf1ae48)
          • hbase-common/src/main/resources/hbase-default.xml
          • hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-1.0 #629 (See https://builds.apache.org/job/HBase-1.0/629/ ) Revert " HBASE-12746 [1.0.0RC0] Distributed Log Replay is on ( HBASE-12577 was insufficient) (Jeffrey Zhong)" (stack: rev d5f63f122131a45b24b5e3839d4b59a578652b2d) hbase-common/src/main/resources/hbase-default.xml hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java HBASE-12746 [1.0.0RC0] Distributed Log Replay is on ( HBASE-12577 was insufficient) (Jeffrey Zhong) (stack: rev a32f485fecbcf7e1aa7159ac2fa9851c1bf1ae48) hbase-common/src/main/resources/hbase-default.xml hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
          Hide
          enis Enis Soztutar added a comment -

          Closing this issue after 0.99.2 release.

          Show
          enis Enis Soztutar added a comment - Closing this issue after 0.99.2 release.

            People

            • Assignee:
              jeffreyz Jeffrey Zhong
              Reporter:
              enis Enis Soztutar
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development