[HBASE-20727] Persist FlushedSequenceId to speed up WAL split after cluster restart

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 3.0.0-alpha-1

Release Note

This feature adds capability for HM to persist its data structure of region wise last flushed seqId. The flush will be happening to hbase root FS path <habse root dir>/.lastflushedseqids
The flushing interval can be tuned using config 'hbase.master.flushedsequenceid.flusher.interval' which defaults to 3 hrs
This feature can be turned off by making hbase.master.persist.flushedsequenceid.enabled as false.

• Assignee: Allan Yang
• Reporter: Allan Yang
• Created: 13/Jun/18 08:39
• Updated: 16/Jan/21 02:13
• Resolved: 27/Sep/18 10:21

Description

We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in ServerManager to record the latest flushed seqids of regions and stores. So during log split, we can use seqids stored in those maps to filter out the edits which do not need to be replayed. But, those maps are not persisted. After cluster restart or master restart, info of flushed seqids are all lost.
Here I offer a way to persist those info to HDFS, even if master restart, we can still use those info to filter WAL edits and then to speed up replay.

Issue Links

• relates to: HBASE-25506 ServerManager affects MTTR of HMaster