[YUNIKORN-1109] Fix data race in core event handling

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: core - scheduler
• Labels: pull-request-available
• Assignee: Craig Condit
• Reporter: Craig Condit
• Created: 08/Mar/22 18:00
• Updated: 04/May/22 23:52
• Resolved: 08/Mar/22 18:50

Description

This data race was encountered on another PR, but appears to have been caused by YUNIKORN-961:

 

 

==================
WARNING: DATA RACE
Read at 0x00c000155bf0 by goroutine 29:
  reflect.maplen()
      /opt/hostedtoolcache/go/1.16.14/x64/src/runtime/map.go:1360 +0x0
  reflect.Value.Len()
      /opt/hostedtoolcache/go/1.16.14/x64/src/reflect/value.go:1155 +0x2ef
  encoding/json.isEmptyValue()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:344 +0x4ce
  encoding/json.structEncoder.encode()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:750 +0x44b
  encoding/json.structEncoder.encode-fm()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:732 +0xb1
  encoding/json.ptrEncoder.encode()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:944 +0x1c4
  encoding/json.ptrEncoder.encode-fm()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:929 +0x88
  encoding/json.structEncoder.encode()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:761 +0x3c6
  encoding/json.structEncoder.encode-fm()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:732 +0xb1
  encoding/json.ptrEncoder.encode()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:944 +0x1c4
  encoding/json.ptrEncoder.encode-fm()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:929 +0x88
  encoding/json.(*encodeState).reflectValue()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:360 +0x93
  encoding/json.(*encodeState).marshal()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/encode.go:332 +0xd2
  encoding/json.(*Encoder).Encode()
      /opt/hostedtoolcache/go/1.16.14/x64/src/encoding/json/stream.go:206 +0xdd
  go.uber.org/zap/zapcore.(*jsonEncoder).encodeReflected()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/zapcore/json_encoder.go:157 +0xac
  go.uber.org/zap/zapcore.(*jsonEncoder).AddReflected()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/zapcore/json_encoder.go:165 +0x6f
  go.uber.org/zap/zapcore.Field.AddTo()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/zapcore/field.go:159 +0xd75
  go.uber.org/zap/zapcore.addFields()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/zapcore/field.go:199 +0x118
  go.uber.org/zap/zapcore.consoleEncoder.writeContext()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/zapcore/console_encoder.go:131 +0x110
  go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/zapcore/console_encoder.go:110 +0x46f
  go.uber.org/zap/zapcore.(*ioCore).Write()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/zapcore/core.go:86 +0x124
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/zapcore/entry.go:216 +0x1f1
  go.uber.org/zap.(*Logger).Debug()
      /home/runner/go/pkg/mod/go.uber.org/zap@v1.13.0/logger.go:179 +0x9a
  github.com/apache/incubator-yunikorn-core/pkg/scheduler.enqueueAndCheckFull()
      /home/runner/work/incubator-yunikorn-core/incubator-yunikorn-core/pkg/scheduler/scheduler.go:102 +0x444
  github.com/apache/incubator-yunikorn-core/pkg/scheduler.(*Scheduler).HandleEvent()
      /home/runner/work/incubator-yunikorn-core/incubator-yunikorn-core/pkg/scheduler/scheduler.go:96 +0x64
  github.com/apache/incubator-yunikorn-core/pkg/rmproxy.(*RMProxy).RegisterResourceManager.func2()
      /home/runner/work/incubator-yunikorn-core/incubator-yunikorn-core/pkg/rmproxy/rmproxy.go:270 +0xf5

Previous write at 0x00c000155bf0 by goroutine 26:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.16.14/x64/src/runtime/map_faststr.go:202 +0x0
  github.com/apache/incubator-yunikorn-core/pkg/scheduler.(*ClusterContext).SetRMInfos()
      /home/runner/work/incubator-yunikorn-core/incubator-yunikorn-core/pkg/scheduler/context.go:913 +0x899
  github.com/apache/incubator-yunikorn-core/pkg/scheduler.(*ClusterContext).processRMRegistrationEvent()
      /home/runner/work/incubator-yunikorn-core/incubator-yunikorn-core/pkg/scheduler/context.go:181 +0x6a5
  github.com/apache/incubator-yunikorn-core/pkg/scheduler.(*Scheduler).handleRMEvent()
      /home/runner/work/incubator-yunikorn-core/incubator-yunikorn-core/pkg/scheduler/scheduler.go:125 +0x458

This seems to be due to updating the RM build information on the event while simultaneously logging a message about the event from another goroutine.

 

Issue Links

• is caused by: YUNIKORN-961 Expose build and version information in a REST call
• links to: GitHub Pull Request #383