[HDFS-6978] Directory scanner should correctly reconcile blocks on RAM disk

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HDFS-6581
• Fix Version/s: 2.6.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Arpit Agarwal
• Reporter: Arpit Agarwal
• Created: 29/Aug/14 21:29
• Updated: 01/Dec/14 03:08
• Resolved: 13/Sep/14 05:14

Description

It used to be very unlikely that the directory scanner encountered two replicas of the same block on different volumes.

With memory storage, it is very likely to hit this with the following sequence of events:

1. Block is written to RAM disk
2. Lazy writer saves a copy on persistent volume
3. DN attempts to evict the original replica from RAM disk, file deletion fails as the replica is in use.
4. Directory scanner finds a replica on both RAM disk and persistent storage.

The directory scanner should never delete the block on persistent storage.