[SPARK-23157] withColumn fails for a column that is a result of mapped DataSet

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.1
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Henry Robinson
• Reporter: Tomasz Bartczak
• Created: 19/Jan/18 16:24
• Updated: 12/Dec/22 18:11
• Resolved: 01/Feb/18 02:16

Description

Having

case class R(id: String)
val ds = spark.createDataset(Seq(R("1")))

This works:

scala> ds.withColumn("n", ds.col("id"))
res16: org.apache.spark.sql.DataFrame = [id: string, n: string]

but when we map over ds it fails:

scala> ds.withColumn("n", ds.map(a => a).col("id"))
org.apache.spark.sql.AnalysisException: resolved attribute(s) id#55 missing from id#4 in operator !Project [id#4, id#55 AS n#57];;
!Project [id#4, id#55 AS n#57]
+- LocalRelation [id#4]

  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:39)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:91)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:347)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:78)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:78)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:91)
  at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:52)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:67)
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2884)
  at org.apache.spark.sql.Dataset.select(Dataset.scala:1150)
  at org.apache.spark.sql.Dataset.withColumn(Dataset.scala:1905)
  ... 48 elided

Issue Links

• links to: [Github] Pull Request #20429 (henryr)
• links to: [Github] Pull Request #20443 (henryr)