[YARN-2791] Add Disk as a resource for scheduling

• Type: Sub-task
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.5.1
• Component/s: scheduler
• Assignee: Yuliya Feldman
• Reporter: Swapnil Daingade
• Created: 31/Oct/14 21:24
• Updated: 21/Oct/15 02:09

Description

Currently, the number of disks present on a node is not considered a factor while scheduling containers on that node. Having large amount of memory on a node can lead to high number of containers being launched on that node, all of which compete for I/O bandwidth. This multiplexing of I/O across containers can lead to slower overall progress and sub-optimal resource utilization as containers starved for I/O bandwidth hold on to other resources like cpu and memory. This problem can be solved by considering disk as a resource and including it in deciding how many containers can be concurrently run on a node.

Issue Links

• duplicates: YARN-2139 [Umbrella] Support for Disk as a Resource in YARN
• is duplicated by: YARN-2817 Disk drive as a resource in YARN
• is related to: YARN-4276 Per-application-type local dirs for NodeManager