Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-2946

Nifi can skip data in List processors and never ingest it afterwards

    XMLWordPrintableJSON

    Details

    • Flags:
      Important

      Description

      With current design for Listing processors, it seems like Nifi may skip some folder's contents which will never be ingested later.

      Example from FTPTransfer.java
      if (recurse && file.isDirectory()) 
      {
      try 
      {
      listing.addAll(getListing(newFullForwardPath, depth + 1, maxResults - count));
      } 
      catch (final IOException e) 
      {
      logger.error("Unable to get listing from " + newFullForwardPath + "; skipping this subdirectory");
       }
       }
      

      With this design, you can encounter a situation in which you can't list one or more subfolder's contents (for any reason like a sudden time-out) BUT still update the component state timestamps : at next runs previous "forgotten" files will never be pushed down because of their timestamps + they can't be reinjected through an error relationship.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ndup-apache Nicolas Dupont
            • Votes:
              2 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: