[YUNIKORN-2559] DATA RACE: EventStore.Store() and Context.PublishEvents() - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.6.0
Component/s: core - common
Labels:
- pull-request-available

Target Version:

1.6.0

Description

How to reproduce:

In shim, update core version to the latest version (v0.0.0-20240415111844-72540e2b277f)
go mod tidy
Run 'make test > shim-racing-log.txt'

WARNING: DATA RACE
Write at 0x00c003882008 by goroutine 59:
  github.com/apache/yunikorn-core/pkg/events.(*EventStore).Store()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_store.go:59 +0x1aa
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher.func2()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:194 +0x167Previous read at 0x00c003882008 by goroutine 60:
  github.com/apache/yunikorn-k8shim/pkg/cache.(*Context).PublishEvents()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/cache/context.go:1176 +0x97
  github.com/apache/yunikorn-k8shim/pkg/cache.(*AsyncRMCallback).SendEvent()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/cache/scheduler_callback.go:235 +0xec
  github.com/apache/yunikorn-core/pkg/events.(*EventPublisher).StartService.func1()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_publisher.go:60 +0x27dGoroutine 59 (running) created at:
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:183 +0x287
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartService()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:166 +0x2b
  github.com/apache/yunikorn-core/pkg/entrypoint.startAllServicesWithParameters()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:80 +0x9b
  github.com/apache/yunikorn-core/pkg/entrypoint.StartAllServices()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:43 +0x59
  github.com/apache/yunikorn-k8shim/pkg/shim.(*MockScheduler).init()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_mock_test.go:63 +0xad
  github.com/apache/yunikorn-k8shim/pkg/shim.TestApplicationScheduling()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_test.go:60 +0x8c
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1742 +0x44Goroutine 60 (running) created at:
  github.com/apache/yunikorn-core/pkg/events.(*EventPublisher).StartService()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_publisher.go:50 +0xc4
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:203 +0x2b8
  github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartService()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:166 +0x2b
  github.com/apache/yunikorn-core/pkg/entrypoint.startAllServicesWithParameters()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:80 +0x9b
  github.com/apache/yunikorn-core/pkg/entrypoint.StartAllServices()
      /home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:43 +0x59
  github.com/apache/yunikorn-k8shim/pkg/shim.(*MockScheduler).init()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_mock_test.go:63 +0xad
  github.com/apache/yunikorn-k8shim/pkg/shim.TestApplicationScheduling()
      /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_test.go:60 +0x8c
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      /usr/local/go/src/testing/testing.go:1742 +0x44
==================

Root cause:

EventStore.events were read/write by 2 different goroutine.

Goroutine 1: EventPublisher.StartService() call sp.store.CollectEvents(), return reference of es.events[:es.idx]
Goroutine 2: EventSystemImpl.StartServiceWithPublisher() call ec.Store.Store(event) to update es.events[es.idx] = event