[SPARK-7292] Provide operator to truncate lineage without persisting RDD's

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Spark Core
• Target Version/s: 1.5.0
• Assignee: Andrew Or
• Reporter: Patrick Wendell
• Created: 01/May/15 07:36
• Updated: 15/Jul/15 03:37
• Resolved: 15/Jul/15 03:37

Description

Checkpointing exists in Spark to truncate a lineage chain. I've heard requests from some users to allow truncation of lineage in a way that is "cheap" and doesn't serialized and persist the RDD. This is possible if the user is willing to forgo fault tolerance for that RDD (for instance, for shorter running jobs or ones that use a small number of machines). It's pretty easy to allow this so we should look into it for Spark 1.5.

Issue Links

• duplicates: SPARK-1855 Provide memory-and-local-disk RDD checkpointing
• links to: [Github] Pull Request #7279 (andrewor14)