[MNEMONIC-213] Create a mnemonic-spark module

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0-incubating
• Fix Version/s: 0.9.0-incubating
• Component/s: Spark-Integration
• Assignee: Gordon King
• Reporter: Gordon King
• Due: 17/Mar/17
• Created: 03/Mar/17 05:33
• Updated: 10/Jul/17 16:57
• Resolved: 10/Jul/17 16:57

Description

To add a new module for Spark integration, this module would contain all Spark related sub-modules.
In this module, it is required to introduce Scala compiler to build Scala part codes that would bring many changes about Mnemonic dev. environment.

Sub-Tasks

• Construct a workable skeleton sub-project of mnemonic-spark: Closed
• Implement DurableRDD: Closed
• Expose allocator from ObjectCreator for creating auxiliary durable objects: Closed
• Add a test case to verify the summation of Long type dataset & bugfixes: Closed
• Using Scala feature to remove redundant getters and setters: Closed
• Explicitly specify the type of allocator for function argument: Closed
• Class Path is not serializable as a parameter of RDD so fallback to String: Closed
• Make test data generated from Random for unit tests: Closed
• Fix the cleanup mechanism for DurableRDD: Closed
• Add a test case to verify the durable chunk type dataset & bugfixes: Closed
• Add a test case to verify the durable generic object type dataset & bugfixes: Closed
• Add a test case to verify buffer type dataset & bugfixes: Closed