Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-7216

FetchSFTP can't delete or move files upon completion

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.10.0
    • 1.12.0, 1.11.4
    • Core Framework
    • None

    Description

      Possibly similar to NIFI-7177 as they're not able to delete files with the GetSFTP processor, but they're getting a different error than I am.

       

      I'm using the FetchSFTP processor to get data from a third party SFTP server, I don't have a lot of details on version or configuration on their end. I have noticed that neither the "Move File" or "Delete File" Completion Strategy options work.

      When using "Delete File" as the Completion Strategy I get no alert/bulletin from NiFi to show that anything went wrong, however, the original file has not been deleted. I can manually delete the file using an SFTP client with the same username/password that NiFi connects as.

      When using "Move File" as the Completion Strategy I get this warning message when NiFi tries to move the file after completion: 

      2020-03-03 13:59:54,375 WARN [Timer-Driven Process Thread-2] o.a.nifi.processors.standard.FetchSFTP FetchSFTP[id=0344354b-3a49-316e-a571-adcca7b3e70e] Successfully fetched the content for StandardFlowFileRecord[uuid=dea58017-8f3f-4991-91a3-1ff8ca167b6c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1583243711464-8, container=default, section=8], offset=4625, length=7672936],offset=0,name=TEST.csv,size=7672936] from ftp1.XXXXX.com:22/Test/TEST.csv but failed to rename the remote file due to java.io.FileNotFoundException: No such file or directory: 

      As with deletion, I am able manually move (rename) the file using an SFTP client with the same username/password that NiFi connects as.

      However, I will add that I'm only able to delete or move the file PRIOR to NiFi fetching the file. If I try to do it immediately after NiFi has fetched the file, I'll get the exact same "no such file or directory" error from the SFTP server when using my local client. If I wait some arbitrary amount of time after NiFi has fetched the file, I am then able to delete or move the file.

      I've also noticed the below sequence of events:

      1. NiFi lists the files on the SFTP server
      2. NiFi fetches the files on the SFTP server
      3. NiFi attempts to delete or move the files on the SFTP server and fails
      4. I immediately attempt to delete or move the files on the SFTP server and fail
      5. I stop the FetchSFTP processor in NiFi
      6. I immediately attempt to delete or move the files on the SFTP server and it succeeds

      This leads me to believe that there is some sort of locking behavior happening where the fetch operation keeps the file locked for some arbitrary amount of time or until the fetch processor is stopped, which is preventing any other operation from taking place on the file.

      Interestingly enough, I was able to get both the "Delete File" and "Move File" Completion Strategies to work when I spun up a basic SFTP server on my own. It sounds like there is something specific with how locks are handled with this third part SFTP server I'm connecting to and how NiFi uses (or re-uses) SFTP clients/connections.

      As mentioned above, I unfortunately don't have a lot of information about this SFTP server I'm connecting to, the vendor has been very tight-lipped about their configuration for some reason.

      If there's anything else you need me to provide, please let me know.

      Thanks!

      Attachments

        Issue Links

          Activity

            People

              joewitt Joe Witt
              Matt Rodriguez Matt Rodriguez
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: