Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-9735

camel-tarfile throws OutOfMemoryError when splitting large files

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.16.2, 2.17.0
    • Fix Version/s: 2.16.3, 2.17.1, 2.18.0
    • Component/s: camel-tarfile
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      After the change in https://issues.apache.org/jira/browse/CAMEL-9375 I started to get failures with the following stacktrace:

      Caused by: java.lang.OutOfMemoryError: Java heap space
              at java.util.Arrays.copyOf(Arrays.java:3236)
              at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
              at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
              at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
              at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:104)
              at org.apache.camel.util.IOHelper.copy(IOHelper.java:199)
              at org.apache.camel.util.IOHelper.copy(IOHelper.java:169)
              at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:218)
              at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:214)
              at org.apache.camel.converter.stream.StreamCacheConverter.convertToStreamCache(StreamCacheConverter.java:82)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243)
              at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
              at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:293)
              at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168)
              at org.apache.camel.dataformat.tarfile.TarIterator.createNextMessage(TarIterator.java:121)
              at org.apache.camel.dataformat.tarfile.TarIterator.tryAdvanceToNext(TarIterator.java:100)
              at org.apache.camel.dataformat.tarfile.TarIterator.hasNext(TarIterator.java:78)
              at org.apache.camel.processor.Splitter$SplitterIterable$1.hasNext(Splitter.java:163)
              at org.apache.camel.processor.Splitter.updateNewExchange(Splitter.java:256)
      

      I.e. using StreamCache makes the whole entry go into memory and since they in my usecase are several gigabytes it doesn't work very well. I realize that this might be caused since I don't set any StreamCachingStrategy on my context but reading the corresponding doc part I get the impression that the default settings should be that it will spool to disk after reading 128 kbyte.

      However, I really don't want it to do that neither as that would 1) be a significant overhead to write gigabytes of data to a temp file and 2) I might not even have that much space where my temp directory is located. Instead the functionality that I want is that the tarfile entries are streamed and written directly to the destination files without any unnecessary caching in between.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                davsclaus Claus Ibsen
                Reporter:
                sverker Sverker Abrahamsson
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: