[YUNIKORN-2574] totalPartitionResource should not be mutated with AddTo/SubFrom

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0, 1.5.0
• Fix Version/s: 1.5.1, 1.6.0
• Component/s: core - scheduler
• Labels: pull-request-available
• Target Version: 1.5.1, 1.6.0
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 22/Apr/24 14:28
• Updated: 23/Apr/24 19:57
• Resolved: 23/Apr/24 19:57

Description

There is a potential data race in PartitionContext: the field totalPartitionResource is mutated in place. The problem is that the method GetTotalPartitionResource() does not clone it.

func (pc *PartitionContext) GetTotalPartitionResource() *resources.Resource {
	pc.RLock()
	defer pc.RUnlock()

	return pc.totalPartitionResource
}

In general, we should prefer the immutable approach for variables like this, just like in objects.Queue:

func (sq *Queue) IncAllocatedResource(alloc *resources.Resource, nodeReported bool) error {
	// check this queue: failure stops checks if the allocation is not part of a node addition
	newAllocated := resources.Add(sq.allocatedResource, alloc)    <----  New object
        [ ... removed ... ]
	sq.Lock()
	defer sq.Unlock()
	// all OK update this queue
	sq.allocatedResource = newAllocated
	sq.updateAllocatedResourceMetrics()
	return nil
}

// incPendingResource increments pending resource of this queue and its parents.
func (sq *Queue) incPendingResource(delta *resources.Resource) {
	// update the parent
	if sq.parent != nil {
		sq.parent.incPendingResource(delta)
	}
	// update this queue
	sq.Lock()
	defer sq.Unlock()
	sq.pending = resources.Add(sq.pending, delta)     <---- New object
        sq.updatePendingResourceMetrics()
}

Issue Links

• links to: GitHub Pull Request #853