[HDFS-1804] Add a new block-volume device choosing policy that looks at free space

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.4-alpha
• Fix Version/s: 2.1.0-beta
• Component/s: datanode
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Reviewed

Release Note

There is now a new option to have the DN take into account available disk space on each volume when choosing where to place a replica when performing an HDFS write. This can be enabled by setting the config "dfs.datanode.fsdataset.volume.choosing.policy" to the value "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".

• Assignee: Aaron Myers
• Reporter: Harsh J
• Created: 04/Apr/11 18:06
• Updated: 27/Aug/13 22:07
• Resolved: 06/Apr/13 04:11

Description

HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.

An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.

Issue Links

• relates to: HDFS-1312 Re-balance disks within a Datanode
• relates to: HDFS-1120 Make DataNode's block-to-device placement policy pluggable