[CAMEL-11104] Performance issue in camel sftp

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.15.2
• Component/s: camel-sftp

Environment

Performance Enviroment

• Estimated Complexity: Unknown
• Assignee: Unassigned
• Reporter: Ayush Dixit
• Created: 04/Apr/17 07:32
• Updated: 04/Apr/17 07:49
• Resolved: 04/Apr/17 07:35

Description

We have implemented a camel route where we are having camel sftp producer to transfer a files to remote SFTP location but on performance testing on client environment and on our local environment we have observed degradation in the time for transferring files to remote SFTP location. Please find the detailed analysis below. The we tried the various test in our local environment. In each test we put around 22 files on camel file consumer and each file took below time to write the file. PFB details

========================================================================================
When target directory having 20,000 files. Camel sftp producer took around 1 minute 43 second to a transfer file
========================================================================================

DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.FileConsumer> About to process file: GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt] using exchange: Exchange[ST30PERFORPMAJAR17020726.txt]
INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> Picked IOP response file : ST30PERFORPMAJAR17020726.txt from IOP FTP directory
INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> Sending IOP response file : ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[ST30PERFORPMAJAR17020726.txt]

DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.GenericFileConverter> Read file /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt (no charset)
DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> About to store file: QST30PERFORPMAJAR17020726.txt using stream: java.io.BufferedInputStream@54a89ff5
DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> Took 0.658 seconds (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP client returned: true

DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.RemoteFileProducer> Wrote [icoms/1/QST30PERFORPMAJAR17020726.txt] to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> Renaming file: icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ST30PERFORPMAJAR17020726.txt
INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt successfully sent to ICOMS FTP directory

=======================================================================================
When target directory having 40,000 files. Camel sftp producer took around 3 minute 17 second to transfer file
=======================================================================================

DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.FileConsumer> About to process file: GenericFile[/data/users/slvm02/smp53/domains/v
mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] using exchange: Exchange[SX30RPMAJAR1702483756.txt]
INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> Picked IOP response file :
SX30RPMAJAR1702483756.txt from IOP FTP directory
INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> Sending IOP response file
: SX30RPMAJAR1702483756.txt to ICOMS FTP directory
DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPer
Poll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt]

DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.GenericFileConverter> Read file /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt (no charset)
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> About to store file: QSX30RPMAJAR1702483756.txt using stream: java.io.BufferedInputStream@7954cfa8
DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> Took 0.659 seconds (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client returned: true
DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.RemoteFileProducer> Wrote [icoms/1/QSX30RPMAJAR1702483756.txt] to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> Renaming file: icoms/1/QSX30RPMAJAR1702483756.txt to: icoms/1/SX30RPMAJAR1702483756.txt
INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt successfully sent to ICOMS FTP directory

====================================================================================================
Similarly when we achieved the files from target directory. It took around 6 sec.It seems like there is a performance issue with camel sftp component. Does it list the files in target directory which is taking time.
===================================================================================================

PFB the producer route which we set up
sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser