[FLINK-2157] Create evaluation framework for ML library

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Component/s: Library / Machine Learning
• Labels: ML pull-request-available
• Assignee: Theodore Vasiloudis
• Reporter: Till Rohrmann
• Created: 04/Jun/15 09:10
• Updated: 28/Feb/19 22:58
• Resolved: 28/Feb/19 22:57

Description

Currently, FlinkML lacks means to evaluate the performance of trained models. It would be great to add some Evaluators which can calculate some score based on the information about true and predicted labels. This could also be used for the cross validation to choose the right hyper parameters.

Possible scores could be F score [1], zero-one-loss score, etc.

Resources
[1] http://en.wikipedia.org/wiki/F1_score

Issue Links

• blocks: FLINK-2269 Add Receiver operating characteristic (ROC) curve evaluation
• is related to: FLINK-2108 Add score function for Predictors
• is required by: FLINK-1723 Add cross validation for model evaluation
• is required by: FLINK-2260 Have a complete model evaluation and selection framework for FlinkML
• links to: GitHub Pull Request #1849
• links to: GitHub Pull Request #1849