Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-14947

camel-sftp: check for existance of remote directory using ls is very slow

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.25.1
    • Fix Version/s: 3.3.0
    • Component/s: camel-ftp
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      In CAMEL-12733 the check for existing directories used by autoCreate was changed from ChannelSftp.cd to ChannelSftp.ls to fix a Windows specific issue.

      cd is a remote operation while ls will actually load the full directory listing and load it over the wire. This happens for every file that are to be written. If the remote directory has a lot of files this has a heavy performance penalty.

      In our case the directory some times contains 50.000+ files. Without autoCreate the writes take a few milliseconds. With autoCreate it takes up to 10 seconds due to the huge directory listing being generated and loaded over the wire for every single file to be written.

      This can be solved by reverting the directory check to a remote operation such as cd and fixing the Windows specific issue in another way.

      If it turns out that ls has to be used (unlikely) please note that there is another variant that takes a selector that enables you to abort the actual loading of the listing:

      ls(String path, LsEntrySelector selector)

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                davsclaus Claus Ibsen
                Reporter:
                chrned Christian Nedregård
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: