[HAWQ-786] Framework to support pluggable formats and file systems

• Type: Wish
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.4.0.0
• Component/s: Storage
• Assignee: Wan Chiyang
• Reporter: Lei Chang
• Created: 08/Jun/16 00:24
• Updated: 26/Jul/18 07:54
• Resolved: 26/Jul/18 07:43

Description

In current HAWQ, two native formats are supported: AO and parquet. Now we want to support ORC. A framework to support naive c/c++ pluggable format is needed to support ORC more easily. And it can also be potentially used for fast external data access.

And there are a lot of requests for supporting S3, Ceph and other file systems, and this is closely related to pluggable formats, so this JIRA is proposing a framework to support both.

Issue Links

• is duplicated by: HAWQ-1270 Plugged storage back-ends for HAWQ
• relates to: HAWQ-864 Support ORC as a native file format
• links to: GitHub Pull Request #701

Sub-Tasks

• Import apache/orc library for further development: Resolved
• Extend orc library to support hdfs located file read.: Resolved
• Implement reading tools for testing HDFS located orc file usage.: Resolved
• Add orc library compiling inside HAWQ build: Open
• Remove apache orc codebase from hawq: Resolved
• Need an switch for the orc support: Open
• Add access interfaces for protocol and format in pluggable storage framework: Closed
• Add Pluggable Storage Dependent Information: Resolved
• Include Pluggable Storage Format Framework in External Table Scan: Resolved
• Include Pluggable Storage Format Framework in External Table Insert: Resolved
• Add demo for pluggable format scan: Resolved
• Add demo for pluggable format insert: Resolved
• Support HDFS protocol using pluggable storage framework: Closed
• Support ORC format using pluggable storage framework: Open
• Support TEXT/CSV format using pluggable storage framework: Open
• Support Hive protocol using pluggable storage framework: Open
• Wrap up pluggable storage framework and HDFS/Hive ORC/TEXT/CSV support: Open