[TINKERPOP-925] Use persisted SparkContext to persist an RDD across Spark jobs.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.2-incubating
• Fix Version/s: 3.1.0-incubating
• Component/s: hadoop
• Assignee: Marko A. Rodriguez
• Reporter: Marko A. Rodriguez
• Created: 27/Oct/15 21:01
• Updated: 02/Nov/15 16:34
• Resolved: 02/Nov/15 15:52

Description

If a provider is using Spark, they are currently forced to have HDFS be used to store intermediate RDD data. However, if they plan on using that data in a GraphComputer "job chain," then they should be able to lookup a .cached() RDD by name.

Create a inputGraphRDD.name and outputGraphRDD.name to make it so that the configuration references SparkContext.getPersitedRDDs().