Details
-
Task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Identify and remove the duplicate tuples in the data. Finding the duplicates requires comparing the pairwise similarity, which is an expensive process. Apply the clustering or blocking techniques to divide the data into partitions and then only compare the pairwise similarity inside partitions.
the builtin could be named as dedup() with parameters like matrix dataset, string similarityMeasure (euclidean, manhattan, cosine e.t.c.), and boolean returnDuplicates (if TRUE, return the duplicate rows only, if FALSE return the original dataset without duplicate rows)