[YARN-45] [Preemption] Scheduler feedback to AM to release containers

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0-beta
• Component/s: resourcemanager
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Carlo Curino
• Reporter: Christopher Douglas
• Created: 25/Aug/12 00:18
• Updated: 13/May/14 21:21
• Resolved: 07/May/13 06:18

Description

The ResourceManager strikes a balance between cluster utilization and strict enforcement of resource invariants in the cluster. Individual allocations of containers must be reclaimed- or reserved- to restore the global invariants when cluster load shifts. In some cases, the ApplicationMaster can respond to fluctuations in resource availability without losing the work already completed by that task (MAPREDUCE-4584). Supplying it with this information would be helpful for overall cluster utilization [1]. To this end, we want to establish a protocol for the RM to ask the AM to release containers.

[1] http://research.yahoo.com/files/yl-2012-003.pdf

Issue Links

• blocks: YARN-397 RM Scheduler api enhancements
• is part of: MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks
• is related to: YARN-291 [Umbrella] Dynamic resource configuration

Sub-Tasks

• Expose preemption warnings in AMRMClient: Open
• Inconsistent picture of how a container was killed when querying RM and NM in case of preemption: Open
• CS queue level preemption should respect user-limits: Open
• Preemption can prevent progress in small queues: Open
• Capacity scheduler preemption policy should respect yarn.scheduler.minimum-allocation-mb when computing resource of queues: Open
• Indicate preemption timout along with the list of containers to AM (preemption message): Open
• Introducing CANCEL_PREEMPTION event to notify Scheduler and AM when a container is no longer to be preempted: Open