[SPARK-17645] Add feature selector methods based on: False Discovery Rate (FDR) and Family Wise Error rate (FWE)

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: ML, MLlib
• Target Version/s: 2.2.0
• Assignee: Peng Meng
• Reporter: Peng Meng
• Shepherd: Yanbo Liang
• Created: 23/Sep/16 07:56
• Updated: 29/Dec/16 15:29
• Resolved: 28/Dec/16 15:02

Description

Univariate feature selection works by selecting the best features based on univariate statistical tests.
FDR and FWE are a popular univariate statistical test for feature selection.

In 2005, the Benjamini and Hochberg paper on FDR was identified as one of the 25 most-cited statistical papers. The FDR uses the Benjamini-Hochberg procedure in this PR. https://en.wikipedia.org/wiki/False_discovery_rate.
In statistics, FWE is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses tests.
https://en.wikipedia.org/wiki/Family-wise_error_rate

We add FDR and FWE methods for ChiSqSelector in this PR, like it is implemented in scikit-learn.
http://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection

Issue Links

• depends upon: SPARK-17704 ChiSqSelector performance improvement.
• depends upon: SPARK-18088 ChiSqSelector FPR PR cleanups
• links to: [Github] Pull Request #15212 (mpjlu)
• links to: [Github] Pull Request #16434 (mpjlu)