[HADOOP-2012] Periodic verification at the Datanode

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Created: 09/Oct/07 00:58
• Updated: 02/May/13 02:29
• Resolved: 16/Jan/08 22:56

Description

Currently on-disk data corruption on data blocks is detected only when it is read by the client or by another datanode. These errors are detected much earlier if datanode can periodically verify the data checksums for the local blocks.

Some of the issues to consider :

• How should we check the blocks ( no more often than once every couple of weeks ?)
• How do we keep track of when a block was last verfied ( there is a .meta file associcated with each lock ).
• What action to take once a corruption is detected
• Scanning should be done as a very low priority with rest of the datanode disk traffic in mind.

Issue Links

• depends upon: HADOOP-1912 Datanode should support block replacement
• is duplicated by: HADOOP-500 Datanode should scan blocks continuously to detect bad blocks / CRC errors