[SPARK-22649] localCheckpoint support in Dataset API

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: PySpark, SQL
• Labels: easyfix
• Target Version/s: 2.3.0
• Assignee: Fernando Pereira Assign to me
• Reporter: Fernando Pereira
• Created: 29/Nov/17 09:03
• Updated: 20/Dec/17 09:37
• Resolved: 20/Dec/17 04:48

Description

localCheckpoints were recently introduced in RDDs as a way of quickly truncate the execution plan by using the caching subsystem (and therefore can not be considered reliable).

I believe it would be of major interest to offer this API in the Dataset API, and similarly in Python Dataframes.
A possible implementation is at PR19805

Issue Links

• is related to: SPARK-22843 R localCheckpoint API
• links to: [Github] Pull Request #19805 (ferdonline)