[YUNIKORN-2559] DATA RACE: EventStore.Store() and Context.PublishEvents()

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: core - common
• Labels: pull-request-available
• Target Version: 1.6.0
• Assignee: Yu-Lin Chen
• Reporter: Yu-Lin Chen
• Created: 16/Apr/24 05:47
• Updated: 17/Apr/24 11:35
• Resolved: 17/Apr/24 11:35

Description

How to reproduce:

1. In shim, update core version to the latest version (v0.0.0-20240415111844-72540e2b277f)
2. go mod tidy
3. Run 'make test > shim-racing-log.txt'

 

WARNING: DATA RACE
Write at 0x00c003882008 by goroutine 59:
  github.com/apache/yunikorn-core/pkg/events.(*EventStore).Store()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_store.go:59 +0x1aa
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher.func2()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:194 +0x167Previous read at 0x00c003882008 by goroutine 60:
  github.com/apache/yunikorn-k8shim/pkg/cache.(*Context).PublishEvents()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/cache/context.go:1176 +0x97
  github.com/apache/yunikorn-k8shim/pkg/cache.(*AsyncRMCallback).SendEvent()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/cache/scheduler_callback.go:235 +0xec
  github.com/apache/yunikorn-core/pkg/events.(*EventPublisher).StartService.func1()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_publisher.go:60 +0x27dGoroutine 59 (running) created at:
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:183 +0x287
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartService()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:166 +0x2b
  github.com/apache/yunikorn-core/pkg/entrypoint.startAllServicesWithParameters()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:80 +0x9b
  github.com/apache/yunikorn-core/pkg/entrypoint.StartAllServices()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:43 +0x59
  github.com/apache/yunikorn-k8shim/pkg/shim.(*MockScheduler).init()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_mock_test.go:63 +0xad
  github.com/apache/yunikorn-k8shim/pkg/shim.TestApplicationScheduling()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_test.go:60 +0x8c
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1742 +0x44Goroutine 60 (running) created at:
  github.com/apache/yunikorn-core/pkg/events.(*EventPublisher).StartService()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_publisher.go:50 +0xc4
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:203 +0x2b8
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartService()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:166 +0x2b
  github.com/apache/yunikorn-core/pkg/entrypoint.startAllServicesWithParameters()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:80 +0x9b
  github.com/apache/yunikorn-core/pkg/entrypoint.StartAllServices()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:43 +0x59
  github.com/apache/yunikorn-k8shim/pkg/shim.(*MockScheduler).init()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_mock_test.go:63 +0xad
  github.com/apache/yunikorn-k8shim/pkg/shim.TestApplicationScheduling()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_test.go:60 +0x8c
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1742 +0x44
================== 

Root cause:

EventStore.events were read/write by 2 different goroutine.

• Goroutine 1: EventPublisher.StartService() call sp.store.CollectEvents(), return reference of es.events[:es.idx]
• Goroutine 2: EventSystemImpl.StartServiceWithPublisher() call ec.Store.Store(event) to update es.events[es.idx] = event

Issue Links

• links to: GitHub Pull Request #844