Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
There are some methods/functions in Yunikorn which are called frequently and often unnecessarily. On a large, busy cluster, eliminating these calls can result in faster scheduling cycle, therefore better throughput.
In the listed cases below, we can re-use a previously computed value and the expensive copy/sort phase can be eliminated completely.
Retrieving node iterators: in baseNodeCollection.getNodeIteratorInternal(), we always clone the tree of sorted nodes, then we build a slice. The node tree is only modified when a node gets a new score (plus node add/removal). By reusing the sorted list, we avoid cloning an *btree.BTree structure and creating []*Node slices.
Queue sorting: only need sorting if the following occurred:
- Allocated resource changed in one of the child queues (most common)
- Pending resource changed from 0 to "n", or from "n" to 0 (affects filtering)
- Child queue got stopped (affects filtering)
- Child queue structure changed on config update
Application sorting: in Queue.TryAllocate() and Queue.TryPlaceholderAllocate(), sortApplications() always runs. In every iteration, it calls Queue.GetCopyOfApps() then proceeds to sort the apps. It only has to run if something relevant happens from the sorting POV:
- Application added/removed
- Ask added to an application
- Ask max priority changed in at least one application
- Allocated resource changed in at least one application
Request sorting: request (ask) sorting is only necessary when the following occurs:
- Ask added
- pendingAskRepeat gets 0 in an ask
Misc: we can have a bunch of other stuff that helps performance.