[YARN-2] Enhance CS to schedule accounting for both memory and cpu cores

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.3-alpha
• Component/s: capacityscheduler, scheduler
• Assignee: Arun Murthy
• Reporter: Arun Murthy
• Created: 08/Jun/12 05:15
• Updated: 02/May/13 02:29
• Resolved: 09/Jan/13 05:17

Description

With YARN being a general purpose system, it would be useful for several applications (MPI et al) to specify not just memory but also CPU (cores) for their resource requirements. Thus, it would be useful to the CapacityScheduler to account for both.

Issue Links

• blocks: MAPREDUCE-4520 Add experimental support for MR AM to schedule CPUs along-with memory
• blocks: YARN-326 Add multi-resource scheduling to the fair scheduler
• blocks: YARN-8 Add more unit tests for CPU scheduling in CS
• blocks: MAPREDUCE-4256 Improve resource scheduling
• incorporates: YARN-21 Cleanup - Remove org.apache.hadoop.yarn.server.resourcemanager.resource.Resource
• is depended upon by: YARN-5 Add support for FifoScheduler to schedule CPU along with memory.
• is depended upon by: YARN-294 Fixup YARN-2 naming and other nits in comments
• is related to: YARN-160 nodemanagers should obtain cpu/memory values from underlying OS
• relates to: YARN-3 Add support for CPU isolation/monitoring of containers
• relates to: MAPREDUCE-4257 Support fair-sharing option within a MR2 Capacity Scheduler queue

Sub-Tasks

• Make ContainersMonitor pluggable: Resolved
• Add support for CPU isolation/monitoring of containers: Closed
• Add support for DistributedShell to ask for CPUs along with memory: Closed
• Add cpu information to queue metrics: Resolved