[NIFI-5700] limit the size of nifi's flowfile repository

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.1
• Component/s: Core Framework
• Assignee: Unassigned
• Reporter: Brandon Rhys DeVries
• Created: 15/Oct/18 16:19
• Updated: 15/Oct/18 16:19

Description

Creating this ticket so as not to lose the discussion from the dev list [1].  Internally, our team has been working with the idea of setting FlowFile count and heap usage limits, resulting in first stalls and then stops of writes to the FlowFile repository.  Yes, stopping writes to the FlowFile repository isn't good... however, "overfilling" the FlowFile repository can potentially be bad to the point of being unrecoverable.  Putting this safeguard in place may mean requiring manual intervention to get data moving again, but it should prevent the system from failing entirely.

[1] http://apache-nifi-developer-list.39713.n7.nabble.com/Is-there-a-configuration-to-limit-the-size-of-nifi-s-flowfile-repository-td18554.html