[NIFI-11636] ParquetReader buffers up to 2 GB of content into heap unnecessarily

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0-M1, 1.22.0
• Component/s: Extensions
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 02/Jun/23 17:20
• Updated: 02/Jun/23 19:37
• Resolved: 02/Jun/23 19:37

Description

The Parquet Record Reader uses the NiFiSeekableInputStream. Because Parquet requires reading the footer first, this class is intended to use mark/reset so that we can read the footer and then reset back to the beginning.

To achieve this, it calls InputStream.mark(Integer.MAX_VALUE) which will buffer up to 2 GB onto heap. However, the underlying InputStream is the ContentClaimInputStream. The ContentClaimInputStream has smarts built into it to allow resetting without having to buffer content into memory. In particular, if you read over the limit provided and then call reset it will close the InputStream and open a new InputStream from the beginning of the FlowFIle content and seek to the desired offset.

Because of this, we don't need to use InputStream.mark(Integer.MAX_VALUE) and can instead use InputStream.mark(8192) or some similarly small value.

Issue Links

• links to: GitHub Pull Request #7334