[NEMO-351] Empowering Nemo with fast I/O using Apache Crail

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.3
• Labels: gsoc2019 pull-request-available
• Assignee: Haeyoon Cho
• Reporter: Jeongyoon Eo
• Created: 07/Mar/19 05:23
• Updated: 11/Nov/22 08:49
• Resolved: 07/Sep/20 08:46

Description

In the era of 100Gbps network and emerging storage devices like NVMe Flash and SCM, it would be great if we can empower Nemo with them.

Apache Crail is a modular I/O framework for data processing systems like Spark, via which the system can leverage fast network and emerging storage devices. By integrating Crail with Nemo, we can test interesting resource environments like resource disaggregation.

Issue Links

• is blocked by: NEMO-366 DirectByteArrayOutputStream bug in SerializedPartition
• links to: GitHub Pull Request #219