[SPARK-1855] Provide memory-and-local-disk RDD checkpointing

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.5.0
• Component/s: MLlib, Spark Core
• Target Version/s: 1.5.0
• Assignee: Andrew Or
• Reporter: Xiangrui Meng
• Created: 15/May/14 21:36
• Updated: 03/Aug/15 17:59
• Resolved: 03/Aug/15 17:59

Description

Checkpointing is used to cut long lineage while maintaining fault tolerance. The current implementation is HDFS-based. Using the BlockRDD we can create in-memory-and-local-disk (with replication) checkpoints that are not as reliable as HDFS-based solution but faster.

It can help applications that require many iterations.

Issue Links

• is duplicated by: SPARK-7292 Provide operator to truncate lineage without persisting RDD's
• relates to: SPARK-1006 MLlib ALS gets stack overflow with too many iterations
• links to: [Github] Pull Request #7279 (andrewor14)