[YARN-5220] Scheduling of OPPORTUNISTIC containers through YARN RM

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-beta1
• Component/s: resourcemanager

Release Note

This extends the centralized YARN RM in to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion.
This way, users can use OPPORTUNISTIC containers to improve the cluster's utilization, without needing to enable distributed scheduling.

• Assignee: Konstantinos Karanasos
• Reporter: Konstantinos Karanasos
• Created: 08/Jun/16 22:21
• Updated: 20/Sep/18 18:12
• Resolved: 04/Nov/17 17:59

Description

In YARN-2882, we introduced the notion of OPPORTUNISTIC containers, along with the existing GUARANTEED containers of YARN.
OPPORTUNISTIC containers are allowed to be queued at the NMs (YARN-2883), and are executed as long as there are available resources at the NM. Moreover, they are of lower priority than the GUARANTEED containers, that is, they can be preempted for a GUARANTEED container to start its execution.

In YARN-2877, we introduced distributed scheduling in YARN, and enabled OPPORTUNISTIC containers to be scheduled exclusively by distributed schedulers.

In this JIRA, we are proposing to extend the centralized YARN RM in order to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion.
This way, users can use OPPORTUNISTIC containers to improve the cluster's utilization, without the need to enable distributed scheduling.

This JIRA is also related to YARN-1011 that introduces the over-commitment of resources, scheduling additional OPPORTUNISTIC containers to the NMs based on the currently used resources and not based only on the allocated resources.

Issue Links

• blocks: YARN-5415 Add support for NodeLocal and RackLocal OPPORTUNISTIC requests
• relates to: YARN-5688 Make allocation of opportunistic containers asynchronous
• relates to: YARN-5823 Update NMTokens in case of requests with only opportunistic containers
• relates to: YARN-5542 Scheduling of opportunistic containers
• relates to: YARN-5646 Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
• requires: YARN-2882 Add an OPPORTUNISTIC ExecutionType
• requires: YARN-2883 Queuing of container requests in the NM

Sub-Tasks

• Refactor DistributedScheduling framework to pull out common functionality: Resolved
• Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests: Resolved
• Refactor TestOpportunisticContainerAllocation to extend TestAMRMClient: Resolved
• Fix Opportunistic Allocation to set the correct value of Node Http Address: Resolved