Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
0.90.0
-
None
Description
There is a very corner case when bad things could happen(ie data loss):
1) RS #1 is going to roll its HLog - not yet created the new one, old one will get no more writes
2) RS #1 enters GC Pause of Death
3) Master lists HLog files of RS#1 that is has to split as RS#1 is dead, starts splitting
4) RS #1 wakes up, created the new HLog (previous one was rolled) and appends an edit - which is lost
The following seems like a possible solution:
1) Master detects RS#1 is dead
2) The master renames the /hbase/.logs/<regionserver name> directory to something else (say /hbase/.logs/<regionserver name>-dead)
3) Add mkdir support (as opposed to mkdirs) to HDFS - so that a file create fails if the directory doesn't exist. Dhruba tells me this is very doable.
4) RS#1 comes back up and is not able create the new hlog. It restarts itself.
Attachments
Attachments
Issue Links
- depends upon
-
HDFS-1554 Append 0.20: New semantics for recoverLease
- Closed
- incorporates
-
HBASE-4744 Remove @Ignore for testLogRollAfterSplitStart
- Closed
- is blocked by
-
HBASE-2437 Refactor HLog splitLog
- Closed
-
HDFS-1554 Append 0.20: New semantics for recoverLease
- Closed
- is depended upon by
-
HBASE-2592 Race between log splitting and log archiving/deleting
- Closed
- is related to
-
HBASE-2593 Race Between Log Splitting and Log Writing
- Closed
- relates to
-
HBASE-4674 splitLog silently fails
- Closed
- requires
-
HDFS-617 Support for non-recursive create() in HDFS
- Closed
-
HADOOP-6840 Support non-recursive create() in FileSystem & SequenceFile.Writer
- Closed
-
HADOOP-6886 LocalFileSystem Needs createNonRecursive API
- Closed