[YARN-1011] [Umbrella] Schedule containers based on utilization of currently allocated containers

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Karthik Kambatla
• Reporter: Arun C Murthy
• Created: 01/Aug/13 18:26
• Updated: 27/Oct/18 18:20

Description

Currently RM allocates containers and assumes resources allocated are utilized.

RM can, and should, get to a point where it measures utilization of allocated containers and, if appropriate, allocate more (speculative?) containers.

Issue Links

• is related to: YARN-8849 DynoYARN: A simulation and testing infrastructure for YARN clusters

Sub-Tasks

• FS should watch node resource utilization and allocate opportunistic containers if appropriate: Resolved
• Common scheduler changes to support scheduler-specific oversubscription implementations: Resolved
• Provide a knob to turn on over-allocation: Resolved
• Expose per-application over-allocation info in the Resource Manager: Resolved
• Add separate NM overallocation thresholds for cpu and memory: Resolved
• Add NM preemption of opportunistic containers when utilization goes high: Resolved
• Add NM support to launch opportunistic containers based on overallocation: Resolved
• Refactor ContainerScheduler to make oversubscription change easier: Resolved
• Add separate NM preemption thresholds for cpu and memory: Resolved
• Crash NM at start time if oversubscription is on but LinuxContainerExcutor or cgroup is off: Resolved
• Add a configuration to cap how much a NM can be overallocated: Resolved
• Fair Scheduler to explicitly promote OPPORTUNISITIC containers locally at the node where they're running: Resolved
• Allow resource request to opt out of oversubscription in Fair Scheduler: Resolved
• Expose per-node over-allocation info in Node Report: Resolved
• Preserve execution type for containers to be increased by AM post YARN-1015: Resolved
• Exclude findbugs warnings on SchedulerNode.numGuaranteedContainers and numOpportunisticContainers: Resolved
• Make sure AM is not scheduled on an opportunistic container: Resolved
• Add queue-level control to allow all applications in a queue to opt-out: Resolved
• Don't start opportunistic containers at container scheduler/finish event with over-allocation: Resolved
• Use aggregate container utilization instead of node utilization to determine resources available for oversubscription: Resolved
• Refactor AbstractYarnScheduler and CapacityScheduler OPPORTUNISTIC container completion codepaths: Resolved
• FairScheduler crashes RM with oversubscription turned on if an application is killed.: Resolved
• Improve debug messages for NM preemption of OPPORTUNISTIC containers: Resolved
• Fair scheduler promotion does not update container execution type and token: Resolved
• Plumb aggregated application resource utilization from the NM to RM: Resolved
• Add documentation of oversubscription: Open
• CS should watch resource utilization of containers and allocate speculative containers if appropriate: Open
• Allow Applications to demand Guaranteed Containers with Capacity Scheduler: Open
• Handle enforceExecutionType == false in AMRMClient: Open
• SnapshotBasedOverAllocationPolicy always caps the amount of memory availabe to 4 GBs: Resolved
• ResourceUtilization cpu is misused in oversubscription as a percentage: Resolved
• Fair Scheduler promotion does not update container execution type when the application is killed: Open
• NM crashes because Fair scheduler promotes a container that has not been pulled by AM: Open