[SPARK-5844] Optimize Pipeline.fit for ParamGrid

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.3.0
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 16/Feb/15 20:25
• Updated: 21/May/19 04:15
• Resolved: 21/May/19 04:15

Description

This issue was brought up by prudenko in this JIRA .

*Proposal*:
When Pipeline.fit is given an array of ParamMaps, it should operate incrementally:

• For each set of parameters applicable to the first PipelineStage,
  • Fit/transform that stage using that set of parameters.
  • For each set of parameters applicable to the second PipelineStage,
    • etc.

This is essentially a depth-first search on the parameters, where each node/level in the search tree is a PipelineStage and each node's child nodes correspond to the set of ParamMaps for that PipelineStage.

This will avoid recomputing intermediate RDDs during model search.

Issue Links

• Is contained by: SPARK-19071 Optimizations for ML Pipeline Tuning