[HBASE-2312] Possible data loss when RS goes into GC pause while rolling HLog

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.90.0
• Fix Version/s: 0.92.0
• Component/s: master, regionserver
• Assignee: Nicolas Spiegelberg
• Reporter: Karthik Ranganathan
• Created: 12/Mar/10 01:19
• Updated: 20/Nov/15 12:42
• Resolved: 02/Nov/11 20:37

Description

There is a very corner case when bad things could happen(ie data loss):

1) RS #1 is going to roll its HLog - not yet created the new one, old one will get no more writes
2) RS #1 enters GC Pause of Death
3) Master lists HLog files of RS#1 that is has to split as RS#1 is dead, starts splitting
4) RS #1 wakes up, created the new HLog (previous one was rolled) and appends an edit - which is lost

The following seems like a possible solution:

1) Master detects RS#1 is dead
2) The master renames the /hbase/.logs/<regionserver name> directory to something else (say /hbase/.logs/<regionserver name>-dead)
3) Add mkdir support (as opposed to mkdirs) to HDFS - so that a file create fails if the directory doesn't exist. Dhruba tells me this is very doable.
4) RS#1 comes back up and is not able create the new hlog. It restarts itself.

Issue Links

• depends upon: HDFS-1554 Append 0.20: New semantics for recoverLease
• incorporates: HBASE-4744 Remove @Ignore for testLogRollAfterSplitStart
• is blocked by: HBASE-2437 Refactor HLog splitLog
• is blocked by: HDFS-1554 Append 0.20: New semantics for recoverLease
• is depended upon by: HBASE-2592 Race between log splitting and log archiving/deleting
• is related to: HBASE-2593 Race Between Log Splitting and Log Writing
• relates to: HBASE-4674 splitLog silently fails
• requires: HDFS-617 Support for non-recursive create() in HDFS
• requires: HADOOP-6840 Support non-recursive create() in FileSystem & SequenceFile.Writer
• requires: HADOOP-6886 LocalFileSystem Needs createNonRecursive API