[YUNIKORN-2392] Concurrent map write in SchedulerCache

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.5.0
• Assignee: Craig Condit
• Reporter: Peter Bacsko
• Created: 05/Feb/24 21:37
• Updated: 20/Mar/24 14:32
• Resolved: 09/Feb/24 02:59

Description

While running the simple performance test with MockScheduler, the following error was detected:

fatal error: concurrent map writes

goroutine 16 [running]:
github.com/apache/yunikorn-k8shim/pkg/cache/external.(*SchedulerCache).removeSchedulingTask(0xc0004261e0, {0xc00d6be600?, 0xc00d6be600?})
	/home/bacskop/repos/yunikorn-k8shim/pkg/cache/external/scheduler_cache.go:593 +0x45
github.com/apache/yunikorn-k8shim/pkg/cache/external.(*SchedulerCache).updatePod(0xc0004261e0, 0xc034a96900)
	/home/bacskop/repos/yunikorn-k8shim/pkg/cache/external/scheduler_cache.go:511 +0x8d1
github.com/apache/yunikorn-k8shim/pkg/cache/external.(*SchedulerCache).UpdatePod(0xc0004261e0, 0x1a?)
	/home/bacskop/repos/yunikorn-k8shim/pkg/cache/external/scheduler_cache.go:468 +0xba
github.com/apache/yunikorn-k8shim/pkg/cache.(*Context).updateYuniKornPod(0xc00012a090, 0xc034a96900)
	/home/bacskop/repos/yunikorn-k8shim/pkg/cache/context.go:306 +0x285
github.com/apache/yunikorn-k8shim/pkg/cache.(*Context).UpdatePod(0xc00012a090, {0xc0001da060?, 0xc0001da120?}, {0x1f92940?, 0xc034a96900?})
	/home/bacskop/repos/yunikorn-k8shim/pkg/cache/context.go:288 +0x2b9
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/home/bacskop/go/pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/controller.go:246
github.com/apache/yunikorn-k8shim/pkg/client.(*MockedAPIProvider).RunEventHandler.func1()
	/home/bacskop/repos/yunikorn-k8shim/pkg/client/apifactory_mock.go:284 +0x5da
created by github.com/apache/yunikorn-k8shim/pkg/client.(*MockedAPIProvider).RunEventHandler in goroutine 13
	/home/bacskop/repos/yunikorn-k8shim/pkg/client/apifactory_mock.go:240 +0xda

We need locking inside SchedulerCache.addSchedulingTask().

Issue Links

• links to: GitHub Pull Request #784