[SPARK-31777] CrossValidator supports user-supplied folds

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: ML
• Assignee: L. C. Hsieh
• Reporter: Xiangrui Meng
• Created: 20/May/20 18:07
• Updated: 16/Jun/20 23:47
• Resolved: 16/Jun/20 23:47

Description

As a user, I can specify how CrossValidator should create folds by specifying a foldCol, which should be integer type with range [0, numFolds). If foldCol is specified, Spark won't do random k-fold split. This is useful if there are custom logics to create folds, e.g., random split by users instead of random splits of events.

This is similar to SPARK-16206, which is for the RDD-based APIs.

Issue Links

• relates to: SPARK-16206 Defining our own folds using CrossValidator
• links to: [Github] Pull Request #28704 (viirya)