[NIFI-3423] List based processors don't support source directories with high file count.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.1
• Component/s: Core Framework
• Assignee: Unassigned
• Reporter: Matthew Clarke
• Created: 31/Jan/17 19:43
• Updated: 29/Sep/17 22:21

Description

NiFi FlowFile attributes/metadata lives in heap. The List based processors return a complete listing from the target and then creates a FlowFile for each File in that returned listing. The FlowFiles being created are not committed to the list processor's success relationship until all have been created. So you end up running out of NiFi JVM heap memory before that can happen when the returned listing is very large.

It would be nice if the list based processors could commit batches (10,000) of FlowFiles at a time from the returned listing instead of trying to commit them all at once to help avoid heap exhaustion.

Issue Links

• is duplicated by: NIFI-4223 Batching in ListFile processor