[SPARK-17866] Dataset.dropDuplicates (i.e., distinct) should not change the output of child plan

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 2.1.0
• Component/s: SQL
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 11/Oct/16 05:33
• Updated: 17/Jan/17 18:44
• Resolved: 17/Jan/17 18:43

Description

We create new Alias with new exprId in Dataset.dropDuplicates now. However it causes problem when we want to select the columns as follows:

val ds = Seq(("a", 1), ("a", 2), ("b", 1), ("a", 1)).toDS()
// ds("_2") will cause analysis exception
ds.dropDuplicates("_1").select(ds("_1").as[String], ds("_2").as[Int])

Issue Links

• breaks: SPARK-19065 dropDuplicates uses the same expression id for Alias and Attribute and breaks attribute replacement
• links to: [Github] Pull Request #15427 (viirya)