[SPARK-12431] add local checkpointing to GraphX

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.5.2
• Component/s: GraphX
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Edward Seidl
• Created: 18/Dec/15 21:57
• Updated: 20/Jan/20 00:05
• Resolved: 21/May/19 04:35

Description

local checkpointing was added to RDD to speed up iterative spark jobs, but this capability hasn't been added to GraphX. Adding localCheckpoint to GraphImpl, EdgeRDDImpl, and VertexRDDImpl greatly improved the speed of a k-core algorithm I'm using (at the cost of fault tolerance, of course).

Issue Links

• is related to: SPARK-5484 Pregel should checkpoint periodically to avoid StackOverflowError
• is related to: SPARK-17877 Can not checkpoint connectedComponents resulting graph
• links to: [Github] Pull Request #13379 (adeandrade)
• links to: GitHub Pull Request #13379