[SPARK-7276] withColumn is very slow on dataframe with large number of columns

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 1.4.0
• Component/s: SQL
• Target Version/s: 1.4.0
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Wenchen Fan
• Reporter: Alexandre CLEMENT
• Created: 30/Apr/15 10:17
• Updated: 13/Mar/21 05:09
• Resolved: 13/May/15 01:19

Description

The code snippet demonstrates the problem.

    import org.apache.spark.sql._
    import org.apache.spark.sql.types._

    val sparkConf = new SparkConf().setAppName("Spark Test").setMaster(System.getProperty("spark.master", "local[4]"))

    val sc = new SparkContext(sparkConf)
    val sqlContext = new SQLContext(sc)
    import sqlContext.implicits._

    val custs = Seq(
      Row(1, "Bob", 21, 80.5),
      Row(2, "Bobby", 21, 80.5),
      Row(3, "Jean", 21, 80.5),
      Row(4, "Fatime", 21, 80.5)
    )

    var fields = List(
      StructField("id", IntegerType, true),
      StructField("a", IntegerType, true),
      StructField("b", StringType, true),
      StructField("target", DoubleType, false))
    val schema = StructType(fields)

    var rdd = sc.parallelize(custs)
    var df = sqlContext.createDataFrame(rdd, schema)

    for (i <- 1 to 200) {
      val now = System.currentTimeMillis
      df = df.withColumn("a_new_col_" + i, df("a") + i)
      println(s"$i -> " + (System.currentTimeMillis - now))
    }

    df.show()

Issue Links

• links to: [Github] Pull Request #5831 (cloud-fan)