[YARN-6828] [Umbrella] Container preemption using OPPORTUNISTIC containers

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Arun Suresh
• Created: 15/Jul/17 16:48
• Updated: 20/Apr/18 18:46

Description

This is based on discussions with kasha and kkaranasos.

Currently, the YARN schedulers selects containers for preemption only in response to a starved queue / app's request. We propose to allow the Schedulers to mark containers that are allocated over queue capacity/fair-share as Opportunistic containers.

This JIRA proposes to allow Schedulers to:

1. Allocate all containers over the configured queue capacity/weight as OPPORTUNISTIC.
2. Auto-promote running OPPORTUNISTIC containers of apps as and when their GUARANTEED containers complete.

Issue Links

• relates to: YARN-8178 [Umbrella] Resource Over-commitment Based on Opportunistic Container Preemption

Sub-Tasks

• Allow Schedulers to return OPPORTUNISTIC containers when queues go over configured capacity: Patch Available
• Promote Opportunistic Containers to Guaranteed containers when Guaranteed containers complete: Open
• SLS NMSimulator support for Opportunistic Container Queuing: Patch Available