[MAPREDUCE-2636] Scheduling over disks horizontally

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: job submission
• Assignee: Unassigned
• Reporter: Evert Lammerts
• Created: 01/Jul/11 08:29
• Updated: 07/Jan/13 22:12

Description

Based on this message: http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201106.mbox/browser

The JT schedules tasks on nodes based on metadata it gets from the NN. The namenode does not know on which disk a block resides. It might happen that on a node running 4 tasks, all read from the same disk. This can affect performance.

An optimization might be to schedule horizontally over disks instead of nodes. Any ideas?