[SPARK-26111] Support ANOVA F-value between label/feature for the continuous distribution feature selection

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: ML
• Assignee: Huaxin Gao
• Reporter: Bihui Jin
• Created: 19/Nov/18 03:13
• Updated: 12/Jan/21 01:09
• Resolved: 15/May/20 15:00

Description

Current only support the selection of categorical features, while there are many requirements to the selection of continuous distribution features.

ANOVA F-value is one way to select features from the continuous distribution and it's important to support it in spark.

Issue Links

• is related to: SPARK-34080 Add UnivariateFeatureSelector to deprecate existing selectors
• links to: GitHub Pull Request #27322

Sub-Tasks

• Support FValueRegressionSelector for continuous features and continuous labels: Resolved
• add FValueRegressionTest: Resolved
• Remove ChiSqSelector dependency on mllib.ChiSqSelectorModel: Resolved
• Add abstract Selector: Resolved
• Add ANOVASelector: Resolved
• add ANOVATest and FValueTest to PySpark: Resolved
• Add ANOVATest example: Resolved
• Add FValueTest Examples: Resolved
• Add ANOVASelector and FValueSelector to PySpark: Resolved
• Add docs and examples for ANOVASelector and FValueSelector: Resolved