[HBASE-10464] Race condition during RS shutdown that could cause data loss

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.89-fb
• Fix Version/s: 0.89-fb
• Component/s: regionserver
• Assignee: Unassigned
• Reporter: Yunfan Zhong
• Created: 04/Feb/14 21:45
• Updated: 17/Jun/22 05:03
• Resolved: 04/Feb/14 21:54

Description

Bug scenario (T* are timestamps, say T1 < T2 < ... < Tn):
1. Master assigns a region to RS at T1
2. RS works on opening the region during T1 to T3
3. In the mean time of opening the region, RS starts to shut down at T2, and dfs client is closed at T5.
4. Regions owned by the RS get closed as a step of RS shutdown except that the newly opened region is online during T3 to T5 and holds some mutations in memory after possible last flush T4.
5. Since master thinks RS has a clean shutdown, there is no log splitting. The HLog was moved to old logs directory naturally.
6. Mutations in memory between T4 to T5 (if T4 does not exist, T3 to T5) are not flushed. They only exist in WAL if it is turned on.

Fix is to prevent region opening from succeeding when the RS is shutting down.

Issue Links

• relates to: HBASE-10514 Forward port HBASE-10466, possible data loss when failed flushes