Details
Critical Description
In SPARK-26111, we introduced a few univariate feature selectors, which share a common set of params. And they are named after the underlying test, which requires users to understand the test to find the matched scenarios. It would be nice if we introduce a single class called UnivariateFeatureSelector that accepts a selection criterion and a score method (string names). Then we can deprecate all other univariate selectors.
For the params, instead of ask users to provide what score function to use, it is more friendly to ask users to specify the feature and label types (continuous or categorical) and we set a default score function for each combo. We can also detect the types from feature metadata if given. Advanced users can overwrite it (if there are multiple score function that is compatible with the feature type and label type combo). Example (param names are not finalized):
selector = UnivariateFeatureSelector(featureCols=["x", "y", "z"], labelCol=["target"], featureType="categorical", labelType="continuous", select="bestK", k=100)
Attachments
Issue Links
- relates to
-
SPARK-26111 Support ANOVA F-value between label/feature for the continuous distribution feature selection
-
- Resolved
-
- links to
1.
|
Hide FValueTest and AnovaTest |
|
Resolved | Ruifeng Zheng |