[NEMO-6] Support RDD caching

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.1
• Labels: common compiler gsoc2018 runtime
• Assignee: Sanha Lee
• Reporter: John Yang
• Created: 02/Mar/18 04:23
• Updated: 23/Nov/18 05:52
• Resolved: 21/Aug/18 03:25

Description

Support RDD caching specified in Spark user programs.
We'll probably have to extend the current block management modules to support cache evictions. Otherwise, trying to cache large data in memory will result in OOMs.

Issue Links

• blocks: NEMO-56 Cache broadcasted data
• blocks: NEMO-15 Run Java Spark ALS in distributed mode
• blocks: NEMO-158 Support Spark SQL Example
• contains: NEMO-139 Logic in the scheduler for appending jobs
• is blocked by: NEMO-62 Support Multiple DAGs Submission in a Single User Program
• is blocked by: NEMO-152 Support Multiple DAGs Execution in a Single User Program
• Is contained by: NEMO-92 Implement the unimplemented transformations/actions & dataset initialization methods for Spark frontend
• is part of: NEMO-143 Release v0.1