[SPARK-10548] Concurrent execution in SQL does not work

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.1, 1.6.0
• Component/s: SQL
• Target Version/s: 1.5.1, 1.6.0
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 10/Sep/15 20:00
• Updated: 06/May/16 20:20
• Resolved: 08/Mar/16 19:01

Description

From the mailing list:

future { df1.count() } 
future { df2.count() } 

java.lang.IllegalArgumentException: spark.sql.execution.id is already set 
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87) 
        at org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1904) 
        at org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1385) 

=== edit ===

Simple reproduction:

(1 to 100).par.foreach { _ =>
  sc.parallelize(1 to 5).map { i => (i, i) }.toDF("a", "b").count()
}

Issue Links

• is related to: TOREE-283 Can only execute one cell with %%SQL
• relates to: SPARK-13747 Concurrent execution in SQL doesn't work with Scala ForkJoinPool
• links to: [Github] Pull Request #8710 (andrewor14)
• links to: [Github] Pull Request #8721 (andrewor14)