[REEF-1765] Building a Parquet Reader for Potential Integrations with Other ML Frameworks

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.16
• Fix Version/s: 0.16
• Component/s: REEF.NET IO
• Labels: features
• Assignee: Unassigned
• Reporter: Shouheng Yi
• Created: 04/Apr/17 17:55
• Updated: 11/Apr/17 18:37
• Resolved: 11/Apr/17 18:11

Description

Parquet file format is very common in some well-known frameworks like Hadoop and Spark. By enabling REEF to read parquet file, we could potentially integrate with those frameworks. Currently we want to only support data of non-nested types with a table-like property. This allows us to transform the data into formats like RDDs, etc.

A draft of ParquetReader is provided here in a PR: https://github.com/apache/reef/pull/1283