[HDFS-7960] The full block report should prune zombie storages even if they're not empty

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.7.0, 2.6.1, 3.0.0-alpha1
• Labels: 2.6.1-candidate
• Target Version/s: 2.7.0
• Assignee: Colin McCabe
• Reporter: Lei (Eddy) Xu
• Created: 19/Mar/15 23:19
• Updated: 30/Aug/16 01:38
• Resolved: 24/Mar/15 05:02

Description

The full block report should prune zombie storages even if they're not empty. We have seen cases in production where zombie storages have not been pruned subsequent to HDFS-7575. This could arise any time the NameNode thinks there is a block in some old storage which is actually not there. In this case, the block will not show up in the "new" storage (once old is renamed to new) and the old storage will linger forever as a zombie, even with the HDFS-7596 fix applied. This also happens with datanode hotplug, when a drive is removed. In this case, an entire storage (volume) goes away but the blocks do not show up in another storage on the same datanode.

Issue Links

• breaks: HDFS-10301 BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
• is depended upon by: HDFS-7961 Trigger full block report after hot swapping disk
• is related to: HDFS-7979 Initialize block report IDs with a random number
• relates to: HDFS-6830 BlockInfo.addStorage fails when DN changes the storage for a block replica