Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16606

Misleading warning for SparkContext.getOrCreate "WARN SparkContext: Use an existing SparkContext, some configuration may not take effect."

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.1.0
    • 2.1.0
    • Spark Core
    • None

    Description

      SparkContext.getOrCreate should really be checking whether the code gets the already-created instance or creating a new one.

      Just a nit-pick: the warning message should also be "Using..." not "Use"

      scala> sc.version
      res2: String = 2.1.0-SNAPSHOT
      
      scala> sc
      res3: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
      
      scala> SparkContext.getOrCreate
      16/07/18 14:40:31 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.
      res4: org.apache.spark.SparkContext = org.apache.spark.SparkContext@1186374c
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            srowen Sean R. Owen
            jlaskowski Jacek Laskowski
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment