[HDFS-325] DFS should not use round robin policy in determing on which volume (file system partition) to allocate for the next block

• Type: Improvement
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Assignee: Dhruba Borthakur
• Reporter: Runping Qi
• Created: 23/Oct/07 17:53
• Updated: 27/Oct/09 19:30

Description

When multiple file system partitions are configured for the data storage of a data node,
it uses a strict round robin policy to decide which partition to use for writing the next block.
This may result in anormaly cases in which the blocks of a file are not evenly distributed across
the partitions. For example, when we use distcp to copy files with each node have 4 mappers running concurrently,
those 4 mappers are writing to DFS at about the same rate. Thus, it is possible that the 4 mappers write out
blocks interleavingly. If there are 4 file system partitions configured for the local data node, it is possible that each mapper will
continue to write its blocks on to the same file system partition.

A simple random placement policy will avoid such anormaly cases, and does not have any obvious drawbacks.

Issue Links

• depends upon: HADOOP-2559 DFS should place one replica per rack
• is related to: HADOOP-2437 final map output not evenly distributed across multiple disks
• relates to: HDFS-738 Improve the disk utilization of HDFS
• relates to: HDFS-284 dfs.data.dir syntax needs revamping: multiple percentages and weights