[SPARK-10764] Add optional caching to Pipelines

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Joseph K. Bradley
• Created: 22/Sep/15 23:03
• Updated: 12/Jan/20 23:54
• Resolved: 21/May/19 04:16

Description

We need to explore how to cache DataFrames during the execution of Pipelines. It's a hard problem in general to handle automatically or manually, so we should start with some design discussions about:

• How to control it manually
• Whether & how to handle it automatically
• API changes needed for each

Issue Links

• links to: [Github] Pull Request #17332 (sachintyagi22)
• links to: GitHub Pull Request #17332