Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10548

Concurrent execution in SQL does not work

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.5.1, 1.6.0
    • Component/s: SQL
    • Labels:
      None

      Description

      From the mailing list:

      future { df1.count() } 
      future { df2.count() } 
      
      java.lang.IllegalArgumentException: spark.sql.execution.id is already set 
              at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87) 
              at org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1904) 
              at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1385) 
      

      === edit ===

      Simple reproduction:

      (1 to 100).par.foreach { _ =>
        sc.parallelize(1 to 5).map { i => (i, i) }.toDF("a", "b").count()
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                andrewor14 Andrew Or
                Reporter:
                andrewor14 Andrew Or
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: