[YARN-5990] [Umbrella] Follow-up (phase-2) FairScheduler preemption improvements

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Karthik Kambatla
• Created: 09/Dec/16 22:25
• Updated: 09/Dec/16 22:25

Description

YARN-4752 overhauled FS preemption. This JIRA is to track follow-up minor improvements discussed in the design doc on YARN-4752.

The reason for creating this is primarily for better release notes around YARN-4752.

Sub-Tasks

• FairScheduler: Avoid preempting AM containers: Resolved
• Propagate allowPreemptionFrom flag all the way down to the app: Resolved
• Experiment with using PriorityQueue for storing starved applications: Resolved
• FS Preemption is a trickle for severely starved applications: Resolved
• FairScheduler might not trigger preemption when using DRF: Resolved
• Set UncaughtExceptionHandler for all FairScheduler threads: Resolved
• Verify app starvation under custom preemption thresholds and timeouts: Open
• Preemption should consider more resource-requests of a starved app if the initial set can not be satisfied: Open
• Test whether FS preemption consider child queues over fair share if the parent is under fair share: Resolved
• FairScheduler preemption and update should not run concurrently: Resolved
• Expose # of pre-emptions as a queue counter: Resolved
• Identifying starved apps does not need the scheduler writelock: Resolved
• Metric preemptCall is not used in new preemption: Resolved
• FairScheduler: Reserve preempted resources for corresponding applications: Resolved
• Add more restrictions In FairScheduler Preemption: Patch Available
• Shuffle nodes when selecting to be preempted nodes: Patch Available
• It's not reasonable to decide whether app is starved by fairShare: Patch Available
• FairScheduler: FSStarvedApps is not thread safe: Patch Available