[HADOOP-2065] Replication policy for corrupted block - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 0.14.1
Fix Version/s: 0.18.0
Component/s: None
Labels:
None

Hadoop Flags:

Incompatible change
Release Note:
Added "corrupt" flag to LocatedBlock to indicate that all replicas of the block thought to be corrupt.

Description

Thanks to ~~HADOOP-1955~~, even if one of the replica is corrupted, the block should get replicated from a good replica relatively fast.

Created this ticket to continue the discussion from http://issues.apache.org/jira/browse/HADOOP-1955#action_12531162.

2. Delete corrupted source replica

3. If all replicas are corrupt, stop replication.

For (2), it'll be nice if the namenode can delete the corrupted block if there's a good replica on other nodes.

For (3), I prefer if the namenode can still replicate the block.
Before 0.14, if the file was corrupted, users were still able to pull the data and decide if they want to delete those files. (~~HADOOP-2063~~)
In 0.14 and later, we cannot/don't replicate these blocks so they eventually get lost.

To make the matters worse, if the corrupted file is accessed, all the corrupted replicas would be deleted except for one and stay as replication factor of 1 forever.