[NIFI-7557] Cache large/common FlowFile attributes when restoring FlowFile Repository

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Core Framework
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 18/Jun/20 16:52
• Updated: 03/Sep/20 14:22
• Resolved: 03/Sep/20 14:22

Description

When NiFi is restarted, it restores FlowFiles from the repository. Each attribute on a FlowFile is read from disk and put into a HashMap. There are times when a Processor will add a large attribute to every FlowFile that it sees, and this results in using much more heap upon NiFi restart to store FlowFiles than it does while NiFi is running. This is because the Processor holds the value of that FlowFile as a single String object and adds that String to the HashMap of attributes on every FlowFile.

However, on restart, NiFi deserializes a byte stream to come up with the attribute value. As a result, each FlowFile that has that attribute value ends up with its own String object, even though the same value is repeated many times.

As a result, a huge amount of heap may be used on restart, causing NiFi to encounter OOME when attempting to restore the FlowFile Repository.

Issue Links

• links to: GitHub Pull Request #4507