[SPARK-31492] flatten the result dataframe of FValueTest

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: ML
• Assignee: Ruifeng Zheng
• Reporter: Ruifeng Zheng
• Created: 20/Apr/20 03:45
• Updated: 18/Jun/22 11:12
• Resolved: 21/Apr/20 03:09

Description

add a new method

@Since("3.1.0")
def test(
    dataset: DataFrame,
    featuresCol: String,
    labelCol: String,
    flatten: Boolean): DataFrame 

 

Similar to new test method in ChiSquareTest, it will:

1, support df operation on the returned df;

2, make driver no longer a bottleneck when dim is high

 

 

Issue Links

• relates to: SPARK-31301 flatten the result dataframe of tests in stat
• links to: [Github] Pull Request #28268 (zhengruifeng)