[YARN-1197] Support changing resources of an allocated container

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0-beta
• Component/s: api, graceful, nodemanager, resourcemanager
• Assignee: Unassigned
• Reporter: Wangda Tan
• Created: 13/Sep/13 08:51
• Updated: 31/Jan/19 16:54

Description

The current YARN resource management logic assumes resource allocated to a container is fixed during the lifetime of it. When users want to change a resource
of an allocated container the only way is releasing it and allocating a new container with expected size.
Allowing run-time changing resources of an allocated container will give us better control of resource usage in application side

Issue Links

• is depended upon by: SLIDER-938 Add ability to resize containers (Hadoop 2.8+)
• is depended upon by: YARN-896 Roll up for long-lived services in YARN
• is related to: YARN-5085 Add support for change of container ExecutionType
• is related to: YARN-4080 Capacity planning for long running services on YARN
• relates to: MAPREDUCE-7180 Relaunching Failed Containers
• relates to: SPARK-3174 Provide elastic scaling within a Spark application

Sub-Tasks

• Common PB type definitions for container resizing: Closed
• AM-RM protocol changes to support container resizing: Closed
• AM-RM protocol changes to support container resizing: Resolved
• AM-NM protocol changes to support container resizing: Resolved
• ContainerManager implementation to support container resizing: Resolved
• RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing: Resolved
• ContainerImpl changes to support container resizing: Resolved
• ContainerManager recovery for container resizing: Resolved
• Make ContainersMonitor can support change monitoring size of an allocated container in NM side: Resolved
• CapacityScheduler side changes to support increase/decrease container resource.: Resolved
• Support change container resource in RM: Resolved
• Roll back container resource allocation after resource increase token expires: Resolved
• Make AMRMClient support send increase container request and get increased/decreased containers: Resolved
• Make NMClient support change container resources: Resolved
• Add implementations to FairScheduler to support increase/decrease container resource: Patch Available
• Protocol changes in RM side to support change container resource: Resolved
• [YARN-1197] Add increased/decreased container to Allocation: Resolved
• [YARN-1197] Modify ApplicationMasterService to support changing container resource: Resolved
• [YARN-1197] Modify ResourceTrackerService to support passing decreased containers to RMNode: Resolved
• [YARN-1197] Add pullDecreasedContainer API to RMNode which can be used by scheduler to get newly decreased Containers: Resolved
• [YARN-1197] Add methods in FiCaSchedulerNode to support increase/decrease/reserve/unreserve change container requests/results: Resolved
• [YARN-1197] Add APIs in CSQueue to support decrease container resource and unreserve increase request: Resolved
• [YARN-1197] Add implementations to CapacityScheduler to support increase/decrease container resource: Resolved
• Merge YARN-1197 back to trunk: Resolved
• Support changing container cpu resource: Resolved
• Resolve findbugs/javac warnings in YARN-1197 branch: Resolved
• Example of use YARN-1197: In Progress
• Increasing container resource while there is no headroom left will cause ResourceManager to crash: Resolved