[SPARK-7324] Add DataFrame.dropDuplicates

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SQL
• Sprint: Spark 1.5 doc/QA sprint
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 02/May/15 08:19
• Updated: 15/Dec/15 11:24
• Resolved: 12/May/15 02:18

Description

Similar to http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html

def dropDuplicates(): DataFrame

def dropDuplicates(subset: Seq[String]): DataFrame

We can turn this into groupBy(cols).agg(first(...))

Issue Links

• relates to: SPARK-12337 Implement dropDuplicates() method of DataFrame in SparkR
• links to: [Github] Pull Request #5870 (kaka1992)
• links to: [Github] Pull Request #6066 (rxin)