[SPARK-19071] Optimizations for ML Pipeline Tuning

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Bryan Cutler
• Created: 04/Jan/17 00:29
• Updated: 21/May/19 04:14
• Resolved: 21/May/19 04:14

Description

This is a parent task to plan the addition of optimizations in ML tuning for parallel model evaluation and more efficiency with pipelines. They will benefit Crossvalidator and TrainValidationSplit when performing a parameter grid search. The proposal can be broken into 3 steps in order of simplicity:

1. Add ability to evaluate models in parallel.

2. Optimize param grid for pipelines, as described in SPARK-5844

3. Add parallel model evaluation to the optimized pipelines from step 2

See the linked design document for details on the proposed implementation.

Issue Links

• contains: SPARK-5844 Optimize Pipeline.fit for ParamGrid
• supercedes: SPARK-14084 Parallel training jobs in model selection
• links to: Design Doc