[HDFS-8486] DN startup may cause severe data loss

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.1, 2.0.0-alpha
• Fix Version/s: 2.6.1, 2.8.0, 2.7.1, 3.0.0-alpha1
• Component/s: datanode
• Labels: 2.6.1-candidate

Release Note

<!-- markdown -->
Public service notice:
* Every restart of a 2.6.x or 2.7.0 DN incurs a risk of unwanted block deletion.
* Apply this patch if you are running a pre-2.7.1 release.

• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 27/May/15 16:46
• Updated: 06/Jan/17 00:42
• Resolved: 02/Jun/15 18:49

Description

A race condition between block pool initialization and the directory scanner may cause a mass deletion of blocks in multiple storages.

If block pool initialization finds a block on disk that is already in the replica map, it deletes one of the blocks based on size, GS, etc. Unfortunately it always deletes one of the blocks even if identical, thus the replica map must be empty when the pool is initialized.

The directory scanner starts at a random time within its periodic interval (default 6h). If the scanner starts very early it races to populate the replica map, causing the block pool init to erroneously delete blocks.

Issue Links

• is broken by: HDFS-6931 Move lazily persisted replicas to finalized directory on DN startup
• is broken by: HDFS-2560 Refactor BPOfferService to be a static inner class
• relates to: HDFS-8193 Add the ability to delay replica deletion for a period of time
• relates to: HDFS-8521 Add @VisibleForTesting annotation to {{BlockPoolSlice#selectReplicaToDelete}}