Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-10900

OutOfMemoryError with big file in ftp or sftp producer with charset assigned

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Workaround
    • 2.17.5
    • 2.19.0
    • camel-ftp, camel-sftp
    • None

    • Centos 7, Java 1.8

    • Unknown

    Description

      During production of sftp or ftp file with charset assigned I have a OutOfMemory for a file of 500MB.

      I think the problem is at this row:
      org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:875)

      and

      org.apache.camel.component.file.remote.FtpOperations.doStoreFile(FtpOperations.java:573)

      the row is :

      is = new ByteArrayInputStream(exchange.getIn().getMandatoryBody(String.class).getBytes(charset));

      Try to convert the file to a String.

      Full stack trace:
      org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: out/outfile.txt
      at org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:902)
      at org.apache.camel.component.file.remote.SftpOperations.storeFile(SftpOperations.java:817)
      at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)
      at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)
      at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
      at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
      at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:173)
      at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:341)
      at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:168)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681)
      at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:609)
      at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:238)
      at org.apache.camel.processor.Splitter.process(Splitter.java:108)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
      at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
      at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:117)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.FilterProcessor.process(FilterProcessor.java:57)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
      at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
      at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
      at org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:97)
      at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:58)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
      Caused by: org.apache.camel.InvalidPayloadException: No body available of type: java.lang.String but has value: /tmp/myfile.tmp of type: java.io.File on: myfile.tmp. Caused by: Error during type conversion from type: java.lang.String to the required type: java.lang.String with value [Body is file based: /tmp/myfile.tmp] due java.lang.OutOfMemoryError: Java heap space. Exchange[ID-localhost-32802-1487864818750-0-13967030]. Caused by: [org.apache.camel.TypeConversionException - Error during type conversion from type: java.lang.String to the required type: java.lang.String with value [Body is file based: /tmp/myfile.tmp] due java.lang.OutOfMemoryError: Java heap space]
      at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:107)
      at org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:865)
      ... 57 more
      Caused by: org.apache.camel.TypeConversionException: Error during type conversion from type: java.lang.String to the required type: java.lang.String with value [Body is file based: /tmp/myfile.tmp] due java.lang.OutOfMemoryError: Java heap space
      at org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:629)
      at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:190)
      at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:105)
      ... 58 more
      Caused by: org.apache.camel.RuntimeCamelException: java.lang.OutOfMemoryError: Java heap space
      at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1690)
      at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1285)
      at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
      at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:306)
      at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:181)
      ... 59 more
      Caused by: java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:3332)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
      at java.lang.StringBuilder.append(StringBuilder.java:190)
      at org.apache.camel.converter.IOConverter.toString(IOConverter.java:318)
      at org.apache.camel.converter.IOConverter.toString(IOConverter.java:261)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1281)
      at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
      at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:306)
      at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:181)
      at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:105)
      at org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:865)
      at org.apache.camel.component.file.remote.SftpOperations.storeFile(SftpOperations.java:817)
      at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)
      at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)
      at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
      at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
      at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:173)
      at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:341)
      at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:168)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
      at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681)

      Attachments

        Activity

          People

            davsclaus Claus Ibsen
            quinta Luca Quintarelli
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: