[YARN-5593] [Umbrella] Add support for YARN Allocation composed of multiple containers/processes

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Arun Suresh
• Reporter: Arun Suresh
• Created: 30/Aug/16 14:59
• Updated: 24/Dec/17 17:41

Description

Opening this to explicitly call out and track some of the ideas that were discussed in YARN-1040. Specifically the concept of an Allocation which can be used by an AM to start multiple Containers against as long as the sum of resources used by all containers fitsIn() the Resources leased to the Allocation.
This is especially useful for AMs that might want to target certain operations (like upgrade / restart) on specific containers / processes within an Allocation without fear of losing the allocation.

Issue Links

• contains: YARN-4876 Decoupled Init / Destroy of Containers from Start / Stop

Sub-Tasks

• [API] Distinguish Allocation from the container/process/task in the public YARN records: Open
• Create RMAllocation and associated state machine: Open
• Changes in RM to lease and track Allocations instead of Containers: Open
• [API] Extend ApplicationMasterProtocol to use Allocations in addition to Containers: Open
• [API] Extend ContainerManagementProtocol to use Allocations in addition to containers: Open
• Introduce AllocationManager in the NodeManager to handle Allocations: Open
• Changes in the AMRMClient to generate and track containers: Open
• Changes in NodeStatusUpdater and ResourceTrackerService to accommodate Allocations: Open