[HBASE-11094] Distributed log replay is incompatible for rolling restarts

• Type: Sub-task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.99.0, 0.98.4
• Hadoop Flags: Reviewed

Release Note

Configuration setting "hbase.master.distributed.log.replay" is only used by Master(source of truth) and region servers which participate in recovery process will recover region servers either in log splitting or replay mode depending on what setting is told by Master.

When "hbase.master.distributed.log.replay" configuration setting changes, master will wait for all existing log recovery work items drain before it applies the new setting in order not to mix different recovery mode & ease administrator duty to manually wait all recovery work is done and then restart master.

• Assignee: Jeffrey Zhong
• Reporter: Enis Soztutar
• Created: 29/Apr/14 01:48
• Updated: 29/Apr/15 02:17
• Resolved: 15/Jun/14 22:21

Description

0.99.0 comes with dist log replay by default (HBASE-10888). However, reading the code and discussing this with Jeffrey, we realized that the dist log replay code is not compatible with rolling upgrades from 0.98.0 and 1.0.0.

The issue is that, the region server looks at it own configuration to decide whether the region should be opened in replay mode or not. The open region RPC does not contain that info. So if dist log replay is enabled on master, the master will assign the region and schedule replay tasks. If the region is opened in a RS that does not have this conf enabled, then it will happily open the region in normal mode (not replay mode) causing possible (transient) data loss.

Issue Links

• is related to: HBASE-13584 Disable distributed log replay by default for 1.1