[SPARK-14516] Clustering evaluator

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: ML
• Target Version/s: 2.3.0
• Assignee: Marco Gaido
• Reporter: Ruifeng Zheng
• Shepherd: Yanbo Liang
• Created: 10/Apr/16 07:09
• Updated: 27/Mar/23 09:42
• Resolved: 12/Sep/17 10:03

Description

MLlib does not have any general purposed clustering metrics with a ground truth.
In [Scikit-Learn](http://scikit-learn.org/stable/modules/classes.html#clustering-metrics), there are several kinds of metrics for this.
It may be meaningful to add some clustering metrics into MLlib.

This should be added as a ClusteringEvaluator class of extending Evaluator in spark.ml.

Issue Links

• is related to: SPARK-22440 Add Calinski-Harabasz index to ClusteringEvaluator
• is related to: SPARK-23217 Add cosine distance measure to ClusteringEvaluator
• relates to: SPARK-21981 Python API for ClusteringEvaluator
• links to: [Github] Pull Request #18538 (mgaido91)
• links to: [Github] Pull Request #19648 (yanboliang)
• links to: [Github] Pull Request #19676 (mgaido91)