[SPARK-13568] Create feature transformer to impute missing values

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: ML
• Assignee: yuhao yang
• Reporter: Nicholas Pentreath
• Shepherd: Nicholas Pentreath
• Created: 29/Feb/16 14:07
• Updated: 16/Mar/17 10:51
• Resolved: 16/Mar/17 10:51

Description

It is quite common to encounter missing values in data sets. It would be useful to implement a Transformer that can impute missing data points, similar to e.g. Imputer in scikit-learn.

Initially, options for imputation could include mean, median and most frequent, but we could add various other approaches. Where possible existing DataFrame code can be used (e.g. for approximate quantiles etc).

Issue Links

• blocks: SPARK-15040 PySpark impl for ml.feature.Imputer
• blocks: SPARK-15041 adding mode strategy for ml.feature.Imputer for categorical features
• blocks: SPARK-19969 Doc and examples for Imputer
• relates to: SPARK-13639 Statistics.colStats(rdd).mean and variance should handle NaN in the input vectors
• links to: [Github] Pull Request #11601 (hhbyyh)