[SPARK-5484] Pregel should checkpoint periodically to avoid StackOverflowError

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0, 2.3.0
• Component/s: GraphX
• Target Version/s: 2.2.0, 2.3.0
• Assignee: dingding
• Reporter: Ankur Dave
• Created: 29/Jan/15 19:22
• Updated: 31/Oct/17 04:49
• Resolved: 25/Apr/17 18:23

Description

Pregel-based iterative algorithms with more than ~50 iterations begin to slow down and eventually fail with a StackOverflowError due to Spark's lack of support for long lineage chains. Instead, Pregel should checkpoint the graph periodically.

Issue Links

• is related to: SPARK-5561 Generalize PeriodicGraphCheckpointer for RDDs
• is required by: SPARK-10335 GraphX Connected Components fail with large number of iterations
• is required by: SPARK-15042 ConnectedComponents fails to compute graph with 200 vertices (but long paths)
• relates to: SPARK-12431 add local checkpointing to GraphX
• relates to: SPARK-15739 Expose aggregateMessagesWithActiveSet to users.
• links to: [Github] Pull Request #4273 (ankurdave)
• links to: [Github] Pull Request #15125 (dding3)
• links to: [Github] Pull Request #19618 (zhengruifeng)