[SPARK-34855] SparkContext - avoid using local lazy val

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.7
• Fix Version/s: 2.4.8
• Component/s: Spark Core
• Assignee: Li Xian
• Reporter: Li Xian
• Created: 24/Mar/21 09:44
• Updated: 12/Dec/22 18:10
• Resolved: 29/Mar/21 11:14

Description

`org.apache.spark.SparkContext#getCallSite` uses local lazy val for `callsite`. But in scala 2.11, local lazy val need synchronization on the containing object `this` (see https://docs.scala-lang.org/sips/improved-lazy-val-initialization.html#version-6---no-synchronization-on-this-and-concurrent-initialization-of-fields and https://github.com/scala/scala-dev/issues/133 )

`getCallSite` is called at the job submission, and thus will be a bottle neck if we are submitting a large amount of jobs on a single spark session. We observed thread blocked due to this in our load test.

Issue Links

• links to: [Github] Pull Request #31953 (lxian)
• links to: [Github] Pull Request #31988 (viirya)