Description
localCheckpoints were recently introduced in RDDs as a way of quickly truncate the execution plan by using the caching subsystem (and therefore can not be considered reliable).
I believe it would be of major interest to offer this API in the Dataset API, and similarly in Python Dataframes.
A possible implementation is at PR19805
Attachments
Issue Links
- is related to
-
SPARK-22843 R localCheckpoint API
- Resolved
- links to