[YARN-5542] Scheduling of opportunistic containers

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Assignee: Unassigned
• Reporter: Konstantinos Karanasos
• Created: 19/Aug/16 21:41
• Updated: 09/Jan/20 13:51
• Resolved: 09/Jan/20 04:59

Description

This JIRA groups all efforts related to the scheduling of opportunistic containers.
It includes the scheduling of opportunistic container through the central RM (YARN-5220), through distributed scheduling (YARN-2877), as well as the scheduling of containers based on actual node utilization (YARN-1011) and the container promotion/demotion (YARN-5085).

Issue Links

• is related to: YARN-5085 Add support for change of container ExecutionType
• is related to: YARN-5541 Handling of opportunistic containers in the NM
• is related to: YARN-5220 Scheduling of OPPORTUNISTIC containers through YARN RM
• relates to: YARN-5823 Update NMTokens in case of requests with only opportunistic containers
• relates to: YARN-4742 [Umbrella] Enhancements to Distributed Scheduling
• relates to: YARN-2877 Extend YARN to support distributed scheduling

Sub-Tasks

• Add support for NodeLocal and RackLocal OPPORTUNISTIC requests: Resolved
• Limit the number of opportunistic container allocated per AM heartbeat: Resolved
• Simplify opportunistic container parameters and metrics: Resolved
• Skip dispatching opportunistic containers to nodes whose queue is already full: Resolved
• Extend Distributed Shell to support launching job with opportunistic containers: Resolved
• Add a flag in distributed shell to automatically PROMOTE opportunistic containers to guaranteed once they are started: Resolved
• Fix logging for preemption of Opportunistic containers for Guaranteed containers.: Resolved
• Add metrics for queued opportunistic and guaranteed containers.: Resolved
• Handle missing scheduler events in Opportunistic Scheduler.: Resolved
• Opportunistic containers should not be scheduled on Decommissioning nodes.: Resolved
• Add Opportunistic Scheduler metrics in ResourceManager.: Resolved
• Fix Opportunistic Scheduling for node local allocations.: Resolved
• Remove hard coded sleep from Opportunistic Scheduler tests.: Resolved
• Add container allocation latency for Opportunistic Scheduler: Resolved
• Efficient allocation of Opportunistic containers.: Resolved
• Make TestOpportunisticContainerAllocatorAMService more resilient.: Resolved
• Refactor OpportunisticContainerAllocator: Resolved
• Remove unused function from OpportunisticContainerAllocatorAMService: Resolved