[SPARK-2429] Hierarchical Implementation of KMeans

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: MLlib
• Labels: clustering
• Assignee: Yu Ishikawa
• Reporter: R J Nowling
• Shepherd: Xiangrui Meng
• Created: 10/Jul/14 14:16
• Updated: 09/Nov/15 23:07
• Resolved: 09/Nov/15 23:07

Description

Hierarchical clustering algorithms are widely used and would make a nice addition to MLlib. Clustering algorithms are useful for determining relationships between clusters as well as offering faster assignment. Discussion on the dev list suggested the following possible approaches:

• Top down, recursive application of KMeans
• Reuse DecisionTree implementation with different objective function
• Hierarchical SVD

It was also suggested that support for distance metrics other than Euclidean such as negative dot or cosine are necessary.

Issue Links

• duplicates: SPARK-6517 Bisecting k-means clustering
• is related to: SPARK-2966 Add an approximation algorithm for hierarchical clustering to MLlib
• relates to: SPARK-2966 Add an approximation algorithm for hierarchical clustering to MLlib
• links to: [Github] Pull Request #2906 (yu-iskw)