[HBASE-12577] Disable distributed log replay by default - ASF JIRA

Details

Type: Sub-task
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.99.2
Component/s: None
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
Disabled Distributed Log Replay (DLR) as default WAL/hlog splitting mechanism. 1.0.x releases will default to Distributed Log Splitting (DLS) system which is the default for 0.98, 0.96 and 0.94 releases.

DLR improves MTTR (the more regions a server is carrying, the better the improvement. Also, we can start taking writes during WAL/hlog recovery), but rolling upgrade to HBase-1.0.x from earlier releases of 0.98 branch (0.98.0,..,0.98.3) can result in data loss. If you are not doing rolling upgrade, or you are rolling upgrading from a 0.98 greater than or equal to 0.98.4, you can safely turn on distributed log replay by default. Regular upgrade (as opposed to rolling upgrade) is not affected. See ~~HBASE-11094~~ for details.

See ~~HBASE-7006~~ to learn more about DLR. You can enable this feature by setting hbase.master.distributed.log.replay to true in your hbase-site.xml file.

Show
Disabled Distributed Log Replay (DLR) as default WAL/hlog splitting mechanism. 1.0.x releases will default to Distributed Log Splitting (DLS) system which is the default for 0.98, 0.96 and 0.94 releases. DLR improves MTTR (the more regions a server is carrying, the better the improvement. Also, we can start taking writes during WAL/hlog recovery), but rolling upgrade to HBase-1.0.x from earlier releases of 0.98 branch (0.98.0,..,0.98.3) can result in data loss. If you are not doing rolling upgrade, or you are rolling upgrading from a 0.98 greater than or equal to 0.98.4, you can safely turn on distributed log replay by default. Regular upgrade (as opposed to rolling upgrade) is not affected. See HBASE-11094 for details. See HBASE-7006 to learn more about DLR. You can enable this feature by setting hbase.master.distributed.log.replay to true in your hbase-site.xml file.

Description

Distributed log replay is an awesome feature, but due of ~~HBASE-11094~~, the rolling upgrade story from 0.98 is hard to explain / enforce.

The fix for ~~HBASE-11094~~ only went into 0.98.4, meaning rolling upgrades from 0.98.4- might lose data during the upgrade.

I feel no matter how much documentation / warning we do, we cannot prevent users from doing rolling upgrades from 0.98.4- to 1.0. And we do not want to inconvenience the user by requiring a two step rolling upgrade.

Thus I think we should disable dist log replay for 1.0, and re-enable it again for 1.1 (if rolling upgrade from 0.98 is not supported).

ie. undo: ~~HBASE-10888~~

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending

Attachments

HBASE-12567.patch

01/Dec/14 04:19

1 kB

Enis Soztutar
HBASE-12567.patch

26/Nov/14 21:41

1 kB

Jeffrey Zhong

Issue Links

is related to

HBASE-13584 Disable distributed log replay by default for 1.1

Closed

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Sean Busbey added a comment - 25/Nov/14 19:05

+1, though I'd rather see the non-rolling-upgrade barrier move to 2.0.

Show

Sean Busbey added a comment - 25/Nov/14 19:05 +1, though I'd rather see the non-rolling-upgrade barrier move to 2.0.

Hide

Permalink

Jeffrey Zhong added a comment - 26/Nov/14 21:41

Submit for QA run. Thanks.

Show

Jeffrey Zhong added a comment - 26/Nov/14 21:41 Submit for QA run. Thanks.

Hide

Permalink

Hadoop QA added a comment - 26/Nov/14 23:30

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12683925/HBASE-12567.patch
against master branch at commit 57b38339db4683aab01cc92d9b0acac367b9ff36.
ATTACHMENT ID: 12683925

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 checkstyle. The applied patch does not increase the total number of checkstyle errors

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 lineLengths. The patch does not introduce lines longer than 100

+1 site. The mvn site goal succeeds with this patch.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHCM
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
org.apache.hadoop.hbase.client.TestFromClientSide

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//console

This message is automatically generated.

Show

Hadoop QA added a comment - 26/Nov/14 23:30 -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12683925/HBASE-12567.patch against master branch at commit 57b38339db4683aab01cc92d9b0acac367b9ff36. ATTACHMENT ID: 12683925 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor org.apache.hadoop.hbase.client.TestFromClientSide Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11845//console This message is automatically generated.

Hide

Permalink

stack added a comment - 26/Nov/14 23:34

TestHCM is a known issue. The other fails should be fixed just now.

Show

stack added a comment - 26/Nov/14 23:34 TestHCM is a known issue. The other fails should be fixed just now.

Hide

Permalink

Lars Hofhansl added a comment - 26/Nov/14 23:58

Hmm... It seems OK to me to make an exception and make it clear. 0.98.4. Distributed Log Replay is too cool to have to off by default.
I forget now, is it on by default in 0.98 (I know, I know, I could just check)?

Show

Lars Hofhansl added a comment - 26/Nov/14 23:58 Hmm... It seems OK to me to make an exception and make it clear. 0.98.4. Distributed Log Replay is too cool to have to off by default. I forget now, is it on by default in 0.98 (I know, I know, I could just check)?

Hide

Permalink

Enis Soztutar added a comment - 01/Dec/14 04:18

We have never turned it on for 0.98. 0.99.0 and 0.99.1 shipped with it being on, but they are non-production releases. ~~HBASE-10888~~ is the first attempt to turn it on by default AFAIK.

Show

Enis Soztutar added a comment - 01/Dec/14 04:18 We have never turned it on for 0.98. 0.99.0 and 0.99.1 shipped with it being on, but they are non-production releases. HBASE-10888 is the first attempt to turn it on by default AFAIK.

Hide

Permalink

Enis Soztutar added a comment - 01/Dec/14 04:19

Retry hadoopqa.

Show

Enis Soztutar added a comment - 01/Dec/14 04:19 Retry hadoopqa.

Hide

Permalink

Hadoop QA added a comment - 01/Dec/14 06:10

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12684344/HBASE-12567.patch
against master branch at commit 5985b485943e4664b83fad2f2cfd089e6031b062.
ATTACHMENT ID: 12684344

+1 @author. The patch does not contain any @author tags.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 javadoc. The javadoc tool appears to have generated 182 warning messages.

+1 checkstyle. The applied patch does not increase the total number of checkstyle errors

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 lineLengths. The patch does not introduce lines longer than 100

+1 site. The mvn site goal succeeds with this patch.

+1 core tests. The patch passed unit tests in .

Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//console

This message is automatically generated.

Show

Hadoop QA added a comment - 01/Dec/14 06:10 -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12684344/HBASE-12567.patch against master branch at commit 5985b485943e4664b83fad2f2cfd089e6031b062. ATTACHMENT ID: 12684344 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 182 warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/checkstyle-aggregate.html Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11877//console This message is automatically generated.

Hide

Permalink

Lars Hofhansl added a comment - 01/Dec/14 18:16

Thanks Enis Soztutar. Agreed then, let's have it off for 1.0.

Show

Lars Hofhansl added a comment - 01/Dec/14 18:16 Thanks Enis Soztutar . Agreed then, let's have it off for 1.0.

Hide

Permalink

stack added a comment - 01/Dec/14 18:17

Lets write fat note in upgrade section that if you are on 0.98.4+ or clean setup, that you SHOULD enable it.

Show

stack added a comment - 01/Dec/14 18:17 Lets write fat note in upgrade section that if you are on 0.98.4+ or clean setup, that you SHOULD enable it.

Hide

Permalink

Enis Soztutar added a comment - 01/Dec/14 22:21

Pushed to branch-1. Thanks Jeffrey.

Show

Enis Soztutar added a comment - 01/Dec/14 22:21 Pushed to branch-1. Thanks Jeffrey.

Hide

Permalink

stack added a comment - 01/Dec/14 23:37

Added note to the 1.0 upgrade section on DLR.

Show

stack added a comment - 01/Dec/14 23:37 Added note to the 1.0 upgrade section on DLR.

Hide

Permalink

Hudson added a comment - 01/Dec/14 23:44

FAILURE: Integrated in HBase-1.0 #525 (See https://builds.apache.org/job/HBase-1.0/525/)
~~HBASE-12577~~ Disable distributed log replay by default (Jeffrey Zhong) (enis: rev 5610732173dcbbe9bc2c82170780f096385274cd)

hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java

Show

Hudson added a comment - 01/Dec/14 23:44 FAILURE: Integrated in HBase-1.0 #525 (See https://builds.apache.org/job/HBase-1.0/525/ ) HBASE-12577 Disable distributed log replay by default (Jeffrey Zhong) (enis: rev 5610732173dcbbe9bc2c82170780f096385274cd) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java

Hide

Permalink

Enis Soztutar added a comment - 01/Dec/14 23:51

Thanks Stack.

Show

Enis Soztutar added a comment - 01/Dec/14 23:51 Thanks Stack.

Hide

Permalink

stack added a comment - 22/Dec/14 22:31

See ~~HBASE-12746~~. More needed disabling DLR

Show

stack added a comment - 22/Dec/14 22:31 See HBASE-12746 . More needed disabling DLR

Hide

Permalink

Hudson added a comment - 27/Dec/14 19:45

FAILURE: Integrated in HBase-1.0 #614 (See https://builds.apache.org/job/HBase-1.0/614/)
~~HBASE-12746~~ [1.0.0RC0] Distributed Log Replay is on (~~HBASE-12577~~ was insufficient) (Jeffrey Zhong) (stack: rev 2df74fbd4a85de1e5325ba6ad8595f2081238e29)

hbase-common/src/main/resources/hbase-default.xml
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java

Show

Hudson added a comment - 27/Dec/14 19:45 FAILURE: Integrated in HBase-1.0 #614 (See https://builds.apache.org/job/HBase-1.0/614/ ) HBASE-12746 [1.0.0RC0] Distributed Log Replay is on ( HBASE-12577 was insufficient) (Jeffrey Zhong) (stack: rev 2df74fbd4a85de1e5325ba6ad8595f2081238e29) hbase-common/src/main/resources/hbase-default.xml hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java

Hide

Permalink

Hudson added a comment - 03/Jan/15 21:00

SUCCESS: Integrated in HBase-1.0 #629 (See https://builds.apache.org/job/HBase-1.0/629/)
Revert "~~HBASE-12746~~ [1.0.0RC0] Distributed Log Replay is on (~~HBASE-12577~~ was insufficient) (Jeffrey Zhong)" (stack: rev d5f63f122131a45b24b5e3839d4b59a578652b2d)

hbase-common/src/main/resources/hbase-default.xml
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
~~HBASE-12746~~ [1.0.0RC0] Distributed Log Replay is on (~~HBASE-12577~~ was insufficient) (Jeffrey Zhong) (stack: rev a32f485fecbcf7e1aa7159ac2fa9851c1bf1ae48)
hbase-common/src/main/resources/hbase-default.xml
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java

Show

Hudson added a comment - 03/Jan/15 21:00 SUCCESS: Integrated in HBase-1.0 #629 (See https://builds.apache.org/job/HBase-1.0/629/ ) Revert " HBASE-12746 [1.0.0RC0] Distributed Log Replay is on ( HBASE-12577 was insufficient) (Jeffrey Zhong)" (stack: rev d5f63f122131a45b24b5e3839d4b59a578652b2d) hbase-common/src/main/resources/hbase-default.xml hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java HBASE-12746 [1.0.0RC0] Distributed Log Replay is on ( HBASE-12577 was insufficient) (Jeffrey Zhong) (stack: rev a32f485fecbcf7e1aa7159ac2fa9851c1bf1ae48) hbase-common/src/main/resources/hbase-default.xml hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java

Hide

Permalink

Enis Soztutar added a comment - 21/Feb/15 23:46

Closing this issue after 0.99.2 release.

Show

Enis Soztutar added a comment - 21/Feb/15 23:46 Closing this issue after 0.99.2 release.

People

Assignee:

Jeffrey Zhong

Reporter:

Enis Soztutar

Votes:

0 Vote for this issue

Watchers:

11 Start watching this issue

Dates

Created:

25/Nov/14 19:02

Updated:

29/Apr/15 02:18

Resolved:

01/Dec/14 22:21

Development

Agile

View on Board