[NEMO-33] Support Apache Arrow

• Type: Wish
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: data-layer data-processing gsoc2018 in-memory java
• Assignee: Unassigned
• Reporter: Wonook
• Created: 13/Mar/18 04:41
• Updated: 20/Nov/18 16:31

Description

Apache arrow is a cross-language development platform for in-memory data. Having a common data layer might help Nemo to prevent copying and converting different data formats (serializing and deserializing)

Some references: https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-13534 , https://databricks.com/session/improving-python-and-spark-performance-and-interoperability-with-apache-arrow