Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-12900

Avoid unnecessary directory listing in PutSFTP

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.26.0, 2.0.0-M3
    • None
    • None

    Description

      The processor PutSFTP is based on PutFileTransfer.
      Before an actual upload takes place, potential conflicts (e.g. existing file) are identified and resolved using identifyAndResolveConflictFile(...).
      As part of this process, information on the target file is retrieved using FileTransfer.getRemoteFileInfo(...).
      In case of PutSFTP this is implemented by SFTPTransfer.

      The implementation of getRemoteFileInfo executes ls on the target directory path. In case there are a lot of files inside the remote directory, e.g. >10.000 files, the listing reduces the performance of PutSFTP significantly.

      Instead of a listing on the directory, file information should be retrieved using either ls or stat on the target file directly.

      Attachments

        Issue Links

          Activity

            People

              EndzeitBegins endzeit
              EndzeitBegins endzeit
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2.5h
                  2.5h