[SPARK-9950] Wrong Analysis Error for grouping/aggregating on struct fields

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Wenchen Fan
• Reporter: Michael Armbrust
• Created: 13/Aug/15 21:18
• Updated: 17/Aug/15 18:36
• Resolved: 17/Aug/15 18:36

Description

Spark 1.4:

import org.apache.spark.sql.functions._
val df = Seq(("x", (1,1)), ("y", (2, 2))).toDF("a", "b")
df.groupBy("b._1").agg(sum("b._2")).collect()

df: org.apache.spark.sql.DataFrame = [a: string, b: struct<_1:int,_2:int>]
res0: Array[org.apache.spark.sql.Row] = Array([1,1], [2,2])

Spark 1.5

org.apache.spark.sql.AnalysisException: expression 'b' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() if you don't care which value you get.;
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.org$apache$spark$sql$catalyst$analysis$CheckAnalysis$class$$anonfun$$checkValidAggregateExpression$1(CheckAnalysis.scala:110)

Issue Links

• links to: [Github] Pull Request #8222 (cloud-fan)