[NIFI-12241] Add Processors Supporting Efficient Parquet Splitting

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.25.0, 2.0.0-M2
• Component/s: Extensions
• Labels: feature performance pull-request-available
• Assignee: Rajmund Takacs
• Reporter: Rajmund Takacs
• Created: 18/Oct/23 08:12
• Updated: 20/Jan/24 02:58
• Resolved: 10/Dec/23 22:22

Description

SplitParquet processor that expects as input a FlowFile with Parquet content and would take as parameter a number of records as the split configuration.

The processor would generate X flow files with unmodified content and would add attributes with the offsets required to read the group of rows in the flowfile's content.

Then the Parquet Reader would be improved to accept optional flow file attributes containing the information so that the reader can only read the required part of the data.

Instead of having something like

X -> SplitRecord (Parquet / JSON) -> ...

It'd be something like

X -> SplitParquet -> ConvertRecord (Parquet / JSON) -> ...

The goal here is to increase the overall efficiency of this operation for extremely large Parquet files (hundreds of GBs). With the second approach, it could leverage multi-threading for processing a single file.

SplitParquet processor should also have a property (true/false) to write zero-content flow files. The existing FetchParquet processor should be enhanced to accept the flow file attributes for giving offsets. It'd give something like

X -> SplitParquet -> FetchParquet (JSON Writer) -> ...

This way, a load balanced connection could be used between SplitParquet and FetchParquet in order to distribute the work across the nodes (without transferring a lot of data across the nodes of the cluster).

Issue Links

• links to: GitHub Pull Request #7893