We experienced a data loss situation that due to double failures.
One is transient disk failure with edits logs and the other is corrupted fstime.
Here is the detail:
1. NameNode has 2 edits directory (say edit0 and edit1)
2. During an update to edit0, there is a transient disk failure,
making NameNode bump the fstime and mark edit0 as stale
and continue working with edit1.
3. NameNode is shut down. Now, and unluckily fstime in edit0
is corrupted. Hence during NameNode startup, the log in edit0
is replayed, hence data loss.
This bug was found by our Failure Testing Service framework:
For questions, please email us: Thanh Do (firstname.lastname@example.org) and
Haryadi Gunawi (email@example.com)