[NEMO-16] Implement collection of data from executor to client

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.1
• Labels: common compiler pull-request-available runtime
• Assignee: Wonook
• Reporter: Wonook
• Created: 02/Mar/18 04:46
• Updated: 27/Nov/18 06:55
• Resolved: 28/Jun/18 07:08

Description

Our current implementation simply serializes and writes files from the executor, and the client reads the data to collect the RDD data. This should be handled more properly.
Please refer to edu.snu.nemo.compiler.frontend.spark.core.java.SparkFrontendUtils and edu.snu.nemo.compiler.frontend.spark.transform.CollectTransform for more information.

Issue Links

• is blocked by: NEMO-103 Communication between Driver and Client
• links to: GitHub Pull Request #56