[SPARK-19940] FPGrowthModel.transform should skip duplicated items

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: ML
• Assignee: Maciej Szymkiewicz
• Reporter: Maciej Szymkiewicz
• Created: 13/Mar/17 22:55
• Updated: 16/May/17 09:53
• Resolved: 14/Mar/17 14:35

Description

Due to misplaced distinct FPGrowthModel.transform generates duplicated items in the "prediction":

scala> val data = spark.read.text("data/mllib/sample_fpgrowth.txt").select(split($"value", "\\s+").alias("features")) 
data: org.apache.spark.sql.DataFrame = [features: array<string>]

scala> val data = spark.read.text("data/mllib/sample_fpgrowth.txt").select(split($"value", "\\s+").alias("features")) 
data: org.apache.spark.sql.DataFrame = [features: array<string>]

scala> fpm.transform(Seq(Array("t", "s")).toDF("features")).show(1, false)
+--------+---------------------+
|features|prediction           |
+--------+---------------------+
|[t, s]  |[y, x, z, x, y, x, z]|
+--------+---------------------+

Issue Links

• is related to: SPARK-14503 spark.ml Scala API for FPGrowth
• links to: [Github] Pull Request #17283 (zero323)