[SPARK-15383] Support appending new columns from other DataFrames

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.6.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Lijie Xu
• Created: 18/May/16 13:24
• Updated: 18/May/16 14:05
• Resolved: 18/May/16 14:05

Description

It is common to add column(s) from other DataFrame(s) as follows. However, current withColumn() function only supports appending a new column from the same DataFrame.

    val df1 = sc.makeRDD(1 to 5).toDF("a")
    val df2 = sc.makeRDD(10 to 15).toDF("b")
    val df = df1.withColumn("b", df2("b")) // Exception

Exception in thread "main" org.apache.spark.sql.AnalysisException: resolved attribute(s) b#3 missing from a#1 in operator !Project a#1,b#3 AS b#4;

Issue Links

• duplicates: SPARK-7460 Provide DataFrame.zip (analog of RDD.zip) to merge two data frames