[SPARK-3625] In some cases, the RDD.checkpoint does not work

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.0.2, 1.1.0
• Component/s: Spark Core
• Target Version/s: 1.0.3, 1.1.1
• Assignee: Guoqiang Li
• Reporter: Guoqiang Li
• Created: 21/Sep/14 14:52
• Updated: 04/Apr/15 07:45
• Resolved: 06/Dec/14 01:46

Description

The reproduce code:

    sc.setCheckpointDir(checkpointDir)
    val c = sc.parallelize((1 to 1000)).map(_ + 1)
    c.count
    val dep = c.dependencies.head.rdd
    c.checkpoint()
    c.count
    assert(dep != c.dependencies.head.rdd)

This limit is too strict , This makes it difficult to implement SPARK-3623 .

Issue Links

• is duplicated by: SPARK-4094 checkpoint should still be available after rdd actions
• is related to: SPARK-6697 PeriodicGraphCheckpointer is not clear edges.
• is related to: SPARK-3623 Graph should support the checkpoint operation
• links to: [Github] Pull Request #2480 (witgo)