[NIFI-2946] Nifi can skip data in List processors and never ingest it afterwards

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: Core Framework
• Labels: errorhandling listFiles nifi skip
• Environment: All
• Flags: Important
• Assignee: Unassigned
• Reporter: Nicolas Dupont
• Created: 26/Oct/16 08:35
• Updated: 02/Nov/16 08:32

Description

With current design for Listing processors, it seems like Nifi may skip some folder's contents which will never be ingested later.

Example from FTPTransfer.java

if (recurse && file.isDirectory()) 
{
try 
{
listing.addAll(getListing(newFullForwardPath, depth + 1, maxResults - count));
} 
catch (final IOException e) 
{
logger.error("Unable to get listing from " + newFullForwardPath + "; skipping this subdirectory");
 }
 }

With this design, you can encounter a situation in which you can't list one or more subfolder's contents (for any reason like a sudden time-out) BUT still update the component state timestamps : at next runs previous "forgotten" files will never be pushed down because of their timestamps + they can't be reinjected through an error relationship.