[YARN-45] [Preemption] Scheduler feedback to AM to release containers

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0-beta
• Component/s: resourcemanager
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Carlo Curino
• Reporter: Christopher Douglas
• Created: 25/Aug/12 00:18
• Updated: 13/May/14 21:21
• Resolved: 07/May/13 06:18

Description

The ResourceManager strikes a balance between cluster utilization and strict enforcement of resource invariants in the cluster. Individual allocations of containers must be reclaimed- or reserved- to restore the global invariants when cluster load shifts. In some cases, the ApplicationMaster can respond to fluctuations in resource availability without losing the work already completed by that task (MAPREDUCE-4584). Supplying it with this information would be helpful for overall cluster utilization [1]. To this end, we want to establish a protocol for the RM to ask the AM to release containers.

[1] http://research.yahoo.com/files/yl-2012-003.pdf

Issue Links

• blocks: YARN-397 RM Scheduler api enhancements
• is part of: MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks
• is related to: YARN-291 [Umbrella] Dynamic resource configuration

Sub-Tasks

• RM changes to support preemption for FairScheduler and CapacityScheduler: Closed
• CapacityScheduler: support for preemption (using a capacity monitor): Closed
• FairScheduler: support for work-preserving preemption: Closed
• Expose preemption warnings in AMRMClient: Open
• User guide for preemption: Resolved
• Preemption caused Invalid State Event: ACQUIRED at KILLED and caused a task timeout for 30mins: Closed
• Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy: Closed
• observeOnly should be checked before any preemption computation started inside containerBasedPreemptOrKill() of ProportionalCapacityPreemptionPolicy.java: Resolved
• ProportionalCapacitPreemptionPolicy handling of corner cases...: Closed
• Preemption message shouldn’t be created multiple times for same container-id in ProportionalCapacityPreemptionPolicy: Resolved
• ClassCastException is thrown during preemption When a huge job is submitted to a queue B whose resources is used by a job in queueA: Closed
• ContainerExistStatus should define a status for preempted containers: Closed
• Add preemption to CS: Resolved
• Inconsistent picture of how a container was killed when querying RM and NM in case of preemption: Open
• Preemption: Jobs are failing due to AMs are getting launched and killed multiple times: Resolved
• Disable preemption at Queue level: Closed
• CS queue level preemption should respect user-limits: Open
• Preemption of AM containers shouldn't count towards AM failures: Closed
• Preemption can prevent progress in small queues: Open
• Capacity scheduler preemption policy should respect yarn.scheduler.minimum-allocation-mb when computing resource of queues: Open
• Indicate preemption timout along with the list of containers to AM (preemption message): Open
• Introducing CANCEL_PREEMPTION event to notify Scheduler and AM when a container is no longer to be preempted: Open
• CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request: Resolved
• Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates: Resolved
• Do surgical preemption based on reserved container in CapacityScheduler: Resolved