[MAPREDUCE-4256] Improve resource scheduling

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: resourcemanager
• Assignee: Unassigned
• Reporter: Radim Kolar
• Created: 14/May/12 15:47
• Updated: 01/Jan/13 11:10
• Resolved: 01/Jan/13 11:10

Description

Currently resource manager supports only Memory resource during container allocation.

I propose following improvements:

1. add support for CPU utilization. Node CPU used information can be obtained by ResourceCalculatorPlugin.

2. add support for custom resources. In node configuration will be something like:

name=node.resource.GPU, value=1 (node has 1 GPU).

If job will need to use GPU for computation, it will add "GPU=1" requirement to its job config and Resource Manager will allocate container on node with GPU available.

Issue Links

• is blocked by: YARN-2 Enhance CS to schedule accounting for both memory and cpu cores
• relates to: YARN-4 Make ContainersMonitor pluggable
• relates to: YARN-3 Add support for CPU isolation/monitoring of containers