[REEF-1791] Implement reef-runtime-spark

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF
• Assignee: Saikat Kanjilal
• Reporter: Sergiy Matusevych
• Created: 28/Apr/17 00:11
• Updated: 18/Dec/17 22:51

Description

We need to run REEF Tasks on Spark Executors. Ideally, that should require only a few lines of changes in the REEF application configuration. All Spark-related logic must be encapsulated in the reef-runtime-spark module, similar to the existing e.g. reef-runtime-yarn or reef-runtime-local. As a first step, we can have a Java-only solution, but later we'll need to run .NET Tasks on Executors as well.

P.S. Here's a REEF Wiki page with more details: Spark+REEF integration