[HBASE-17437] Support specifying a WAL directory outside of the root directory - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 1.2.4
Fix Version/s: 1.4.0, 1.3.3, 2.0.0
Component/s: Filesystem Integration, wal
Labels:
- patch
- s3

Hadoop Flags:

Reviewed
Release Note:

Hide
This patch adds support for specifying a WAL directory outside of the HBase root directory.

Multiple configuration variables were added to accomplish this:
hbase.wal.dir: used to configure where the root WAL directory is located. Could be on a different FileSystem than the root directory. WAL directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset.

hbase.rootdir.perms: Configures FileSystem permissions to set on the root directory. This is '700' by default.

hbase.wal.dir.perms: Configures FileSystem permissions to set on the WAL directory FileSystem. This is '700' by default.

Show
This patch adds support for specifying a WAL directory outside of the HBase root directory. Multiple configuration variables were added to accomplish this: hbase.wal.dir: used to configure where the root WAL directory is located. Could be on a different FileSystem than the root directory. WAL directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset. hbase.rootdir.perms: Configures FileSystem permissions to set on the root directory. This is '700' by default. hbase.wal.dir.perms: Configures FileSystem permissions to set on the WAL directory FileSystem. This is '700' by default.
Flags:

Patch

Description

Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some FileSystems (such as Amazon S3) don’t support append or consistent writes. These two properties are imperative for the WAL in order to avoid loss of writes. However, StoreFiles don’t necessarily need the same consistency guarantees (since writes are cached locally and if writes fail, they can always be replayed from the WAL).

This JIRA aims to allow users to configure a log directory (for WALs) that is outside of the root directory or even in a different FileSystem. The default value will still put the log directory under the root directory.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

hbase-17437-master.patch
07/Jan/17 02:32
74 kB
Yishan Yang
hbase-17437-branch-1.2.patch
07/Jan/17 02:32
134 kB
Yishan Yang
HBASE-17437.master.001.patch
14/Jan/17 05:40
94 kB
Zach York
HBASE-17437.master.002.patch
14/Jan/17 10:52
94 kB
Zach York
HBASE-17437.master.003.patch
15/Jan/17 15:14
94 kB
Zach York
HBASE-17437.master.004.patch
17/Jan/17 04:07
94 kB
Zach York
HBASE-17437.master.005.patch
30/Jan/17 04:15
103 kB
Zach York
HBASE-17437.master.006.patch
30/Jan/17 04:23
102 kB
Zach York
HBASE-17437.master.007.patch
30/Jan/17 08:14
102 kB
Zach York
HBASE-17437.master.008.patch
30/Jan/17 23:54
100 kB
Zach York
HBASE-17437.master.009.patch
31/Jan/17 00:07
100 kB
Zach York
HBASE-17437.master.010.patch
31/Jan/17 00:23
100 kB
Zach York
HBASE-17437.master.011.patch
31/Jan/17 03:04
96 kB
Zach York
HBASE-17437.master.012.patch
31/Jan/17 08:50
95 kB
Zach York
HBASE-17437.branch-1.001.patch
02/Feb/17 12:39
104 kB
Zach York
HBASE-17437.branch-1.002.patch
07/Feb/17 06:52
128 kB
Zach York
HBASE-17437.branch-1.003.patch
07/Feb/17 08:56
128 kB
Zach York
HBASE-17437.branch-1.004.patch
07/Feb/17 12:25
128 kB
Zach York

Issue Links

Add Link

is related to

HBASE-21755 RS aborts while performing replication with wal dir on hdfs, root dir on s3

Closed

Delete this link

HBASE-21688 Address WAL filesystem issues

Closed

Delete this link

HBASE-17878 java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter when starting HBase with hbase.rootdir on S3