[YARN-1011] [Umbrella] Schedule containers based on utilization of currently allocated containers

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Karthik Kambatla
• Reporter: Arun C Murthy
• Created: 01/Aug/13 18:26
• Updated: 19/Apr/18 23:20

Description

Currently RM allocates containers and assumes resources allocated are utilized.

RM can, and should, get to a point where it measures utilization of allocated containers and, if appropriate, allocate more (speculative?) containers.

Issue Links

• is blocked by: YARN-2882 Add an OPPORTUNISTIC ExecutionType
• is blocked by: YARN-4509 Promote containers from OPPORTUNISTIC to GUARANTEED
• is blocked by: YARN-4885 Aggregate container utilization can be negative
• is blocked by: YARN-5085 Add support for change of container ExecutionType
• is blocked by: YARN-4718 Rename variables in SchedulerNode to reduce ambiguity post YARN-1011
• is duplicated by: YARN-4407 Support Resource oversubscription in YARN scheduler
• is related to: YARN-4597 Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle
• is related to: YARN-3119 Memory limit check need not be enforced unless aggregate usage of all containers is near limit
• is related to: YARN-1197 Support changing resources of an allocated container
• relates to: YARN-5542 Scheduling of opportunistic containers
• relates to: YARN-8178 [Umbrella] Resource Over-commitment Based on Opportunistic Container Preemption
• relates to: YARN-5220 Scheduling of OPPORTUNISTIC containers through YARN RM
• relates to: YARN-2745 Extend YARN to support multi-resource packing of tasks
• relates to: YARN-5202 Dynamic Overcommit of Node Resources - POC
• relates to: YARN-5215 Scheduling containers based on external load in the servers

Sub-Tasks

• Report NM aggregated container resource utilization in heartbeat: Resolved
• CS should watch resource utilization of containers and allocate speculative containers if appropriate: Open
• FS should watch node resource utilization and allocate opportunistic containers if appropriate: Resolved
• Collect memory/cpu usage on the node: Resolved
• ResourceUtilization should be a part of NodeReport API.: Resolved
• ResourceUtilization should be a part of NodeInfo REST API: Resolved
• ResourceUtilization should be a part of yarn node CLI: Resolved
• Plumb resource-utilization info in node heartbeat through to the scheduler: Resolved
• Common scheduler changes to support scheduler-specific oversubscription implementations: Resolved
• Provide a knob to turn on over-allocation: Resolved
• Handle enforceExecutionType == false in AMRMClient: Open
• Expose per-application over-allocation info in the Resource Manager: Resolved
• Add separate NM overallocation thresholds for cpu and memory: Resolved
• Add container type awareness in ResourceHandlers.: Resolved
• Use cgroup to get container resource utilization: Resolved
• Add NM preemption of opportunistic containers when utilization goes high: Resolved
• Add cpu cgroup configurations for opportunistic containers: Resolved
• Add memory cgroup settings for opportunistic containers: Resolved
• Add NM support to launch opportunistic containers based on overallocation: Resolved
• Consolidate NM overallocation thresholds with ResourceTypes: Open
• Refactor ContainerScheduler to make oversubscription change easier: Resolved
• Add separate NM preemption thresholds for cpu and memory: Resolved
• NM overallocation based on over-time rather than snapshot utilization: Open
• Crash NM at start time if oversubscription is on but LinuxContainerExcutor or cgroup is off: Resolved
• Add a configuration to cap how much a NM can be overallocated: Resolved
• Fair Scheduler to explicitly promote OPPORTUNISITIC containers locally at the node where they're running: Resolved
• Add unit test for NM to launch OPPORTUNISTIC container for overallocation: Resolved
• Add opportunity to start containers while periodically checking for preemption: Resolved
• Scheduler-agnostic RM changes support oversubscription: Resolved
• Add unit test for NM to preempt OPPORTUNISTIC containers under high utilization: Resolved
• Allow resource request to opt out of oversubscription in Fair Scheduler: Resolved
• Handle Container ExecType update (Promotion/Demotion) in cgroups resource handlers: Resolved
• Add documentation of oversubscription: Open
• Expose per-node over-allocation info in Node Report: Resolved
• Preserve execution type for containers to be increased by AM post YARN-1015: Resolved
• Exclude findbugs warnings on SchedulerNode.numGuaranteedContainers and numOpportunisticContainers: Resolved
• Make sure AM is not scheduled on an opportunistic container: Resolved
• Add queue-level control to allow all applications in a queue to opt-out: Resolved
• Create another implementation of ContainerScheduler to support NM overallocation: Resolved
• Don't start opportunistic containers at container scheduler/finish event with over-allocation: Resolved
• Fair scheduler promotion does not update container execution type and token: Patch Available