[SPARK-17535] Performance Improvement of Signleton pattern in SparkContext

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.0.0
• Component/s: Spark Core
• Labels: easyfix performance
• Flags: Important
• Assignee: Unassigned
• Reporter: Jianfei Wang
• Created: 14/Sep/16 13:58
• Updated: 15/Sep/16 07:50
• Resolved: 15/Sep/16 07:50

Description

I think the singleton pattern of SparkContext is inefficient if there are many request to get the SparkContext,
So we can write the singleton pattern as below,The second way if more efficient when there are many request to get the SparkContext.

 // the current version
  def getOrCreate(): SparkContext = {
    SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
      if (activeContext.get() == null) {
        setActiveContext(new SparkContext(), allowMultipleContexts = false)
      }
      activeContext.get()
    }
  }
  // by myself
  def getOrCreate(): SparkContext = {
    if (activeContext.get() == null) {
      SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
        if(activeContext.get == null) {
          @volatile val sparkContext = new SparkContext()
          setActiveContext(sparkContext, allowMultipleContexts = false)
        }
      }
      activeContext.get()
    }
  }