[HBASE-23286] Improve MTTR: Split WAL to HFile

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.3.0
• Fix Version/s: 3.0.0-alpha-1, 2.3.0
• Component/s: MTTR

Release Note

Add a new feature to improve MTTR which have 3 steps to failover:
1. Read WAL and write HFile to region’s column family’s recovered.hfiles directory.
2. Open region.
3. Bulkload the recovered.hfiles for every column family.

Compared to DLS(distributed log split), this feature will reduce region open time significantly.

Config hbase.wal.split.to.hfile to true to enable this featue.

• Assignee: Guanghao Zhang
• Reporter: Guanghao Zhang
• Created: 13/Nov/19 10:09
• Updated: 16/Jun/20 03:01
• Resolved: 13/Jan/20 18:34

Description

After HBASE-20724, the compaction event marker is not used anymore when failover. So our new proposal is split WAL to HFile to imporve MTTR. It has 3 steps:

1. Read WAL and write HFile to region’s column family’s recovered.hfiles directory.
2. Open region.
3. Bulkload the recovered.hfiles for every column family.

The design doc was attathed by a google doc. Any suggestions are welcomed.

Issue Links

• breaks: HBASE-23741 Data loss when WAL split to HFile enabled
• breaks: HBASE-23684 NPE HFilesOutputSink
• breaks: HBASE-23742 Document that with split-to-hfile data over the MOB threshold will be treated as normal data
• is related to: HBASE-23633 Find a way to handle the corrupt recovered hfiles
• is related to: HBASE-23634 Enable "Split WAL to HFile" by default
• is related to: HBASE-20991 MTTR
• links to: Design doc
• links to: GitHub Pull Request #820

Sub-Tasks

• BoundedRecoveredHFilesOutputSink should read the table descriptor directly: Resolved
• Invalid StoreFile WARN log message printed for recovered.hfiles directory: Resolved
• Resolve the TODO of FSTableDescriptor's construct method: Resolved
• Add ut for root dir and wal root dir are different: Resolved
• Doc WALSplitter classes: Resolved