[SPARK-5561] Generalize PeriodicGraphCheckpointer for RDDs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.5.0
• Component/s: GraphX, MLlib, Spark Core
• Target Version/s: 1.5.0
• Assignee: Joseph K. Bradley
• Reporter: Joseph K. Bradley
• Shepherd: Xiangrui Meng
• Created: 03/Feb/15 18:47
• Updated: 30/Jul/15 14:56
• Resolved: 30/Jul/15 14:56

Description

PeriodicGraphCheckpointer was introduced for Latent Dirichlet Allocation (LDA), but it could be generalized to work with both Graphs and RDDs. It should be generalized and moved out of MLlib.

(For those who are not familiar with it, it tries to automatically handle persisting/unpersisting and checkpointing/removing checkpoint files in a lineage of Graphs.)

A generalized version might be immediately useful for:

• RandomForest
• Streaming
• GLMs

Issue Links

• relates to: SPARK-5484 Pregel should checkpoint periodically to avoid StackOverflowError
• relates to: SPARK-5955 Add checkpointInterval to ALS
• relates to: SPARK-6684 Add checkpointing to GradientBoostedTrees
• links to: [Github] Pull Request #7728 (jkbradley)