This is a parent task to plan the addition of optimizations in ML tuning for parallel model evaluation and more efficiency with pipelines. They will benefit Crossvalidator and TrainValidationSplit when performing a parameter grid search. The proposal can be broken into 3 steps in order of simplicity:
1. Add ability to evaluate models in parallel.
2. Optimize param grid for pipelines, as described in
3. Add parallel model evaluation to the optimized pipelines from step 2
See the linked design document for details on the proposed implementation.
|Parallel Model Evaluation for ML Tuning: Scala||Resolved|