[SPARK-21087] CrossValidator, TrainValidationSplit should collect all models when fitting: Scala API

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: ML
• Assignee: Weichen Xu
• Reporter: Joseph K. Bradley
• Shepherd: Joseph K. Bradley
• Created: 14/Jun/17 01:57
• Updated: 12/Dec/17 23:03
• Resolved: 15/Nov/17 00:49

Description

We add a parameter whether to collect the full model list when CrossValidator/TrainValidationSplit training (Default is NOT, avoid the change cause OOM)
Add a method in CrossValidatorModel/TrainValidationSplitModel, allow user to get the model list
CrossValidatorModelWriter add a “option”, allow user to control whether to persist the model list to disk.
Note: when persisting the model list, use indices as the sub-model path

Issue Links

• blocks: SPARK-22004 CrossValidator, TrainValidationSplit dump sub models to disk when fitting: Scala API
• links to: [Github] Pull Request #18313 (hhbyyh)
• links to: [Github] Pull Request #19208 (WeichenXu123)
• links to: [Github] Pull Request #19958 (yanboliang)