[HDFS-3702] Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.5.1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: hdfs-client
• Labels: BB2015-05-TBR
• Target Version/s: 2.8.0

Release Note

This patch will attempt to allocate all replicas to remote DataNodes, by adding local DataNode to the excluded DataNodes. If no sufficient replicas can be obtained, it will fall back to default block placement policy, which writes one replica to local DataNode.

• Assignee: Lei (Eddy) Xu
• Reporter: Nicolas Liochon
• Created: 23/Jul/12 10:54
• Updated: 12/Mar/24 12:54
• Resolved: 27/Apr/16 22:12

Description

This is useful for Write-Ahead-Logs: these files are writen for recovery only, and are not read when there are no failures.

Taking HBase as an example, these files will be read only if the process that wrote them (the 'HBase regionserver') dies. This will likely come from a hardware failure, hence the corresponding datanode will be dead as well. So we're writing 3 replicas, but in reality only 2 of them are really useful.

Issue Links

• is duplicated by: HDFS-4946 Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable
• is related to: HDFS-13573 Javadoc for BlockPlacementPolicyDefault is inaccurate
• is related to: HDFS-4946 Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable
• relates to: HBASE-5843 Improve HBase MTTR - Mean Time To Recover
• links to: GitHub Pull Request #176