[YARN-3983] Make CapacityScheduler to easier extend application allocation logic

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 27/Jul/15 21:34
• Updated: 30/Aug/16 01:18
• Resolved: 05/Aug/15 20:52

Description

While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation.

For example, there's a lot of differences between increasing a container and allocating a container:

• Increasing a container doesn't need to check locality delay.
• Increasing a container doesn't need to build/modify a resource request tree (ANY->RACK/HOST).
• Increasing a container doesn't need to check allocation/reservation starvation (see shouldAllocOrReserveNewContainer).
• After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container.

And there're lots of similarities when allocating different types of resources.

• User-limit/queue-limit will be enforced for both of them.
• Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them).

The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization.

Issue Links

• is depended upon by: YARN-1651 CapacityScheduler side changes to support increase/decrease container resource.