Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37503 Improve SparkSession/PySpark SparkSession startup
  3. SPARK-37638

Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • PySpark
    • None

    Description

      >>> ps.range(10)
      21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the static sql configurations will not take effect.
      21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some spark core configurations may not take effect.
      21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the static sql configurations will not take effect.
      21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some spark core configurations may not take effect.
      21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the static sql configurations will not take effect.
      21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some spark core configurations may not take effect.
      21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the static sql configurations will not take effect.
      21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some spark core configurations may not take effect.
      ...
         id
      0   0
      1   1
      2   2
      3   3
      4   4
      5   5
      6   6
      7   7
      8   8
      9   9
      

      Attachments

        Activity

          People

            gurwls223 Hyukjin Kwon
            gurwls223 Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: