[YUNIKORN-1716] Improve the performance of baseNodeCollection.getNodeIteratorInternal() - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.4.0
Component/s: None
Labels:
- pull-request-available

Target Version:

1.4.0

Description

We can improve the performance of baseNodeCollection.getNodeIteratorInternal(). This gets called in every scheduling cycle. In a cluster with thousands of nodes, this has costs. In one of my heap profiles which I collected during some performance testing, this method shows up in the alloc_space heap profile. Not the biggest contributor (around 5%), but it is visible. Therefore, optimizing it has measurable benefits.

Approaches:

caching: only sort nodes if necessary. Unfortunately, every allocation results in new scores, so we probably constantly have to invalidate the cached list.
walking the BTree: the current implementation does not have a stateful iterator, but we can still call BTree.Ascend() with a function parameter, which returns false as soon as we find a suitable node.