[SPARK-22005] CrossValidator, TrainValidationSplit dump sub models to disk when fitting: Python API

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Weichen Xu
• Created: 14/Sep/17 07:19
• Updated: 08/Oct/19 05:43
• Resolved: 08/Oct/19 05:43

Description

In pyspark:
We add a parameter indicating whether to persist models to disk during training (default = off). This will use ML persistence to dump models to a directory so they are available later but do not consume memory.
Note: when persisting the model list, use indices as the sub-model path

Issue Links

• is blocked by: SPARK-21088 CrossValidator, TrainValidationSplit should collect all models when fitting: Python API
• is required by: SPARK-23109 ML 2.3 QA: API: Python API coverage