Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
None
-
None
Description
Currently RDD.checkpoint must be called before any job executed on this RDD, otherwise the `doCheckpoint` will never be called. This is a pitfall we should check this and throw exception (or at least log warning ? ) for such case.
And, if RDD haven't been persisted, doing checkpoint will cause RDD recomputation, because current implementation will run separated job for checkpointing. I think such case it should also print some warning message, remind user to check whether he forgot persist the RDD.
