[HUDI-431] Support Parquet in MOR log files

• Type: New Feature
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: storage-management
• Labels: help-requested pull-request-available
• Story Points: 0
• Assignee: Alexey Kudinkin
• Reporter: sivabalan narayanan

Reviewers

Vinoth Chandar

• Created: 17/Dec/19 23:14
• Updated: 01/Apr/22 04:13
• Resolved: 02/Feb/22 21:28

Description

We have a basic implementation of inline filesystem, to read a file format like Parquet, embedded "inline" into another file.  

https://github.com/apache/hudi/blob/master/hudi-common/src/test/java/org/apache/hudi/common/fs/inline/TestInLineFileSystem.java for sample usage.

 This idea here is to see if we can embed parquet/hfile formats into the Hudi log files, to get columnar reads on the delta log files as well. This helps us speed up query performance, given the log is row based today. Once Inline FS is available, enable parquet logging support with HoodieLogFile. LogFile can expose a writer (essentially ParquetWriter) and users can write records as though writing to parquet files. Similarly on the read path, a reader (parquetReader) will be exposed which the user can use to read data out of it. 

This Jira tracks work to implement such parquet inlining into the log format and have the writer and reader use it. 

 

Issue Links

• is depended upon by: HUDI-466 [Umbrella] Record level, global low-latency index implementation
• links to: GitHub Pull Request #3783
• links to: GitHub Pull Request #3930
• links to: GitHub Pull Request #4333