[YUNIKORN-1715] Yunikorn performance improvements

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0, 1.4.0
• Component/s: core - scheduler
• Labels: release-notes
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 03/May/23 11:59
• Updated: 04/Dec/23 22:22
• Resolved: 04/Dec/23 22:21

Description

There are some methods/functions in Yunikorn which are called frequently and often unnecessarily. On a large, busy cluster, eliminating these calls can result in faster scheduling cycle, therefore better throughput.

In the listed cases below, we can re-use a previously computed value and the expensive copy/sort phase can be eliminated completely.

Retrieving node iterators: in baseNodeCollection.getNodeIteratorInternal(), we always clone the tree of sorted nodes, then we build a slice. The node tree is only modified when a node gets a new score (plus node add/removal). By reusing the sorted list, we avoid cloning an *btree.BTree structure and creating []*Node slices.

Queue sorting: only need sorting if the following occurred:

• Allocated resource changed in one of the child queues (most common)
• Pending resource changed from 0 to "n", or from "n" to 0 (affects filtering)
• Child queue got stopped (affects filtering)
• Child queue structure changed on config update

Application sorting: in Queue.TryAllocate() and Queue.TryPlaceholderAllocate(), sortApplications() always runs. In every iteration, it calls Queue.GetCopyOfApps() then proceeds to sort the apps. It only has to run if something relevant happens from the sorting POV:

• Application added/removed
• Ask added to an application
• Ask max priority changed in at least one application
• Allocated resource changed in at least one application

Request sorting: request (ask) sorting is only necessary when the following occurs:

• Ask added
• pendingAskRepeat gets 0 in an ask

Misc: we can have a bunch of other stuff that helps performance.

Sub-Tasks

• Improve the performance of baseNodeCollection.getNodeIteratorInternal(): Closed
• Improve the performance of Queue.sortQueues(): Closed
• Improve the performance of Queue.sortApplications(): Closed
• Improve the performance of Application.sortRequests(): Closed
• Improve the performance of node.preAllocateCheck(): Closed
• Re-think the usage of *resource.Resource objects: Closed
• Remove string comparison from hasReadyCondition() function: Closed
• Improve the performance of shim side scheduling cycle: Closed
• Add predicate with equivalence cache to improve the scheduler performance: Closed
• Improve the performance of nodeInfoListerImpl.List(): Closed
• Improve the performance of HavePodsWithAffinityList() and HavePodsWithRequiredAntiAffinityList(): Closed
• Don't copy resource object in Node.preAllocateCheck(): Closed
• Avoid resource clone in Application.tryNodes(): Closed
• Application: maintain sorted state of requests instead of sorting: Closed
• Store time in int64 type instead of time.Time: Closed
• Re-use predicate results for reservation: Closed
• Move prefilter plugins out of node predicates to improve performance: Closed
• Improve the performance of tryPlaceholderAllocate: Closed
• Further performance improvements on HavePodsWithAffinityList() and HavePodsWithRequiredAntiAffinityList(): Closed
• Optimize scheduling cycle if there's no placeholder allocation or reservation: Closed