[NIFI-4223] Batching in ListFile processor

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: Configuration
• Labels: features
• Assignee: Unassigned
• Reporter: Mark Heydenrych
• Created: 25/Jul/17 10:56
• Updated: 25/Jul/17 16:00

Description

The ListFile processor builds a full list of all files before sending any flowfiles into the queue to be processed. When the number of files is exceptionally large, building the list takes a very long time and appears to be hanging. It would be best if we could set a batch size e.g. 1000 files, and have the ListFile processor stop listing after the reaching this many files, send the files through to the queue and then resume.

Issue Links

• duplicates: NIFI-3423 List based processors don't support source directories with high file count.