[SPARK-11596] SQL execution very slow for nested query plans because of DataFrame.withNewExecutionId

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.1
• Fix Version/s: 1.6.0
• Component/s: SQL
• Target Version/s: 1.6.0
• Assignee: Yin Huai
• Reporter: Cristian Opris
• Created: 09/Nov/15 17:02
• Updated: 02/Dec/15 12:12
• Resolved: 02/Dec/15 01:19

Description

For nested query plans like a recursive unionAll, withExecutionId is extremely slow, likely because of repeated string concatenation in QueryPlan.simpleString

Test case:

(1 to 100).foldLeft[Option[DataFrame]] (None) { (curr, idx) =>
    println(s"PROCESSING >>>>>>>>>>> $idx")
    val df = sqlContext.sparkContext.parallelize((0 to 10).zipWithIndex).toDF("A", "B")
    val union = curr.map(_.unionAll(df)).getOrElse(df)
    union.cache()
    println(">>" + union.count)
    //union.show()
    Some(union)
  }

Stack trace:

scala.collection.TraversableOnce$class.addString(TraversableOnce.scala:320)
scala.collection.AbstractIterator.addString(Iterator.scala:1157)
scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:286)
scala.collection.AbstractIterator.mkString(Iterator.scala:1157)
scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:288)
scala.collection.AbstractIterator.mkString(Iterator.scala:1157)
org.apache.spark.sql.catalyst.trees.TreeNode.argString(TreeNode.scala:364)
org.apache.spark.sql.catalyst.trees.TreeNode.simpleString(TreeNode.scala:367)
org.apache.spark.sql.catalyst.plans.QueryPlan.simpleString(QueryPlan.scala:168)
org.apache.spark.sql.catalyst.trees.TreeNode.generateTreeString(TreeNode.scala:401)
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$generateTreeString$1.apply(TreeNode.scala:403)
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$generateTreeString$1.apply(TreeNode.scala:403)
scala.collection.immutable.List.foreach(List.scala:318)
org.apache.spark.sql.catalyst.trees.TreeNode.generateTreeString(TreeNode.scala:403)
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$generateTreeString$1.apply(TreeNode.scala:403)
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$generateTreeString$1.apply(TreeNode.scala:403)
scala.collection.immutable.List.foreach(List.scala:318)
org.apache.spark.sql.catalyst.trees.TreeNode.generateTreeString(TreeNode.scala:403)
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$generateTreeString$1.apply(TreeNode.scala:403)
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$generateTreeString$1.apply(TreeNode.scala:403)
scala.collection.immutable.List.foreach(List.scala:318)
org.apache.spark.sql.catalyst.trees.TreeNode.generateTreeString(TreeNode.scala:403)
org.apache.spark.sql.catalyst.trees.TreeNode.treeString(TreeNode.scala:372)
org.apache.spark.sql.catalyst.trees.TreeNode.toString(TreeNode.scala:369)
org.apache.spark.sql.SQLContext$QueryExecution.stringOrError(SQLContext.scala:936)
org.apache.spark.sql.SQLContext$QueryExecution.toString(SQLContext.scala:949)
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:52)
org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1903)
org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1384)
org.apache.spark.sql.DataFrame.count(DataFrame.scala:1402)

Issue Links

• links to: [Github] Pull Request #10079 (yhuai)