[YUNIKORN-1] [Umbrella] Support priority aware scheduling

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: core - scheduler
• Labels: release-notes
• Target Version: 1.2.0
• Assignee: Craig Condit
• Reporter: Weiwei Yang
• Created: 06/Feb/20 06:17
• Updated: 07/Feb/23 22:33
• Resolved: 24/Dec/22 21:31

Description

The scheduler needs to understand app/task priority and make proper decisions while allocating resources. I think we don't need to introduce a very complex model when defining priorities, e.g supporting both queues, app, and task level priorities, it will be hard for users to consume. Instead, we need to find a clean, straightforward approach to handle priorities.

Issue Links

• relates to: YUNIKORN-241 Sort requests within same app first by priority then creation time
• relates to: YUNIKORN-1461 [Umbrella] Preemption support
• supercedes: YUNIKORN-261 Support request sorting policy
• supercedes: YUNIKORN-262 Support app/task priority design doc
• links to: Design Doc (2022)

Sub-Tasks

• Add new queue configuration options: Closed
• Track dynamic priority of applications: Closed
• Track dynamic priority of queues: Closed
• Update queue sorting policies to take into account priorities: Closed
• Update application sorting policies to consider priority: Closed
• Add e2e tests for priority scheduling: Closed
• Add design document to website: Closed
• Document priority feature: Closed
• Ensure dynamic queues are initialized with priority fields: Closed