Cocoon
  1. Cocoon
  2. COCOON-2168

ResourceReader produces Java Heap Overflow when reading a huge resource

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.11, 2.2
    • Fix Version/s: 2.1.12, 2.2.1
    • Component/s: * Cocoon Core
    • Labels:
      None
    • Urgency:
      Normal
    • Affects version (Component):
      Components: Pipeline - 1.0.0-RC1
    • Fix version (Component):
      Components: Pipeline

      Description

      When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.

      As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
      1. ResourceReader.diff
        6 kB
        Felix Knecht
      2. test-case.tar.gz
        2 kB
        Felix Knecht

        Activity

        Hide
        Jörg Heinicke added a comment -
        The allocated memory for the buffer will now be increasing starting from an initial buffer size of 8KB rather than allocating the whole memory for the potentially big buffer (like 1MB) beforehand.
        Show
        Jörg Heinicke added a comment - The allocated memory for the buffer will now be increasing starting from an initial buffer size of 8KB rather than allocating the whole memory for the potentially big buffer (like 1MB) beforehand.
        Hide
        Jörg Heinicke added a comment -
        As discussed in http://marc.info/?t=120473411300003&r=1&w=4 I changed the default buffer size used in Cocoon to 1 MB. This will prevent OOMEs with overly large resources in most circumstances when the user has not configured the buffer size. On the other hand it will hardly affect any "normal" Cocoon pipeline rendering since 1 MB for markup is rather large. The buffer size can still be configured to complete buffering by setting it to -1 explicitly (which used to be the default).
        Show
        Jörg Heinicke added a comment - As discussed in http://marc.info/?t=120473411300003&r=1&w=4 I changed the default buffer size used in Cocoon to 1 MB. This will prevent OOMEs with overly large resources in most circumstances when the user has not configured the buffer size. On the other hand it will hardly affect any "normal" Cocoon pipeline rendering since 1 MB for markup is rather large. The buffer size can still be configured to complete buffering by setting it to -1 explicitly (which used to be the default).
        Hide
        Felix Knecht added a comment -
        It's a configuration problem.
        Show
        Felix Knecht added a comment - It's a configuration problem.
        Hide
        Felix Knecht added a comment -
        After all it turned out that it's a problem of configuration. Thanks to Carsten.

        Last snippet of the mail thread thread http://marc.info/?t=120473411300003&r=1&w=2

          <map:pipeline id="test-nocache" type="noncaching">
             <map:parameter name="outputBufferSize" value="0" />
             <map:match pattern="nocache">
               <map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso" >
                 <map:parameter name="buffer-size" value="8192" />
               </map:read>
             </map:match>
           </map:pipeline>

        The output buffer only specifies the size of the buffer for writing the response. This is not directly related to caching. You might increase performance by buffering.

        The buffer is in both case unlimited because of allowing proper error handling.

        If we don't have it yet, we should add these things to a "tuning cocoon" page. I would turn off infinite buffering in production in all case and set a fixed buffer size (like 8192). For reader pipelines I would turn off buffering completly.
        Show
        Felix Knecht added a comment - After all it turned out that it's a problem of configuration. Thanks to Carsten. Last snippet of the mail thread thread http://marc.info/?t=120473411300003&r=1&w=2   <map:pipeline id="test-nocache" type="noncaching">      <map:parameter name="outputBufferSize" value="0" />      <map:match pattern="nocache">        <map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso" >          <map:parameter name="buffer-size" value="8192" />        </map:read>      </map:match>    </map:pipeline> The output buffer only specifies the size of the buffer for writing the response. This is not directly related to caching. You might increase performance by buffering. The buffer is in both case unlimited because of allowing proper error handling. If we don't have it yet, we should add these things to a "tuning cocoon" page. I would turn off infinite buffering in production in all case and set a fixed buffer size (like 8192). For reader pipelines I would turn off buffering completly.
        Hide
        Felix Knecht added a comment -
        I agree with Joerg that my patch is against the idea of buffering the whole pipeline. OTH (looking at security issues) I think it can be dangerous to accept Overflows by using a default configuration - I don't know how Java behaves in this case.

        That's why I'd prefer another default value than -1 if none is configured (no matter if it's 8192 or 1024^2). But I think it should be a value which doesn't causes the application to fail when run with default settings and using large resource files.
        Show
        Felix Knecht added a comment - I agree with Joerg that my patch is against the idea of buffering the whole pipeline. OTH (looking at security issues) I think it can be dangerous to accept Overflows by using a default configuration - I don't know how Java behaves in this case. That's why I'd prefer another default value than -1 if none is configured (no matter if it's 8192 or 1024^2). But I think it should be a value which doesn't causes the application to fail when run with default settings and using large resource files.
        Hide
        Jörg Heinicke added a comment -
        I've looked into this issue and I'm against your fix. It stands completely against the idea of buffering the whole pipeline content which is done for error handlers. It should be possible for them to reset the OutputStream. Or asked in other words? Why using the non-flushing BufferedOutputStream at all?

        Now how to handle the issue with huge resources? Should be quite easy. From what I see from the code (AbstractProcessingPipeline) it is possible to configure and setup/parameterize a pipeline with "outputBufferSize". This means on both map:pipe and map:pipeline it should be possible to set an actual buffer size. Only if none is set (and it defaults to -1) the non-flushing BufferedOutputStream is used.

        Root sitemap has this example (in map:components/map:pipes section):

             <map:pipe name="noncaching" src="org.apache.cocoon.components.pipeline.impl.NonCachingProcessingPipeline"
                       pool-max="${noncaching-pipeline.pool-max}">
               <!-- parameter name="outputBufferSize" value="8192"/ -->
             </map:pipe>

        The following should also work (in map:pipelines section):

            <map:pipeline>
               <map:parameter name="outputBufferSize" value="8192"/>
               ...
            </map:pipeline>

        We could argue about another default value than -1 though. Something like 1024^2.

        WDYT?
        Show
        Jörg Heinicke added a comment - I've looked into this issue and I'm against your fix. It stands completely against the idea of buffering the whole pipeline content which is done for error handlers. It should be possible for them to reset the OutputStream. Or asked in other words? Why using the non-flushing BufferedOutputStream at all? Now how to handle the issue with huge resources? Should be quite easy. From what I see from the code (AbstractProcessingPipeline) it is possible to configure and setup/parameterize a pipeline with "outputBufferSize". This means on both map:pipe and map:pipeline it should be possible to set an actual buffer size. Only if none is set (and it defaults to -1) the non-flushing BufferedOutputStream is used. Root sitemap has this example (in map:components/map:pipes section):      <map:pipe name="noncaching" src="org.apache.cocoon.components.pipeline.impl.NonCachingProcessingPipeline"                pool-max="${noncaching-pipeline.pool-max}">        <!-- parameter name="outputBufferSize" value="8192"/ -->      </map:pipe> The following should also work (in map:pipelines section):     <map:pipeline>        <map:parameter name="outputBufferSize" value="8192"/>        ...     </map:pipeline> We could argue about another default value than -1 though. Something like 1024^2. WDYT?
        Hide
        Felix Knecht added a comment -
        Patch for ResourceReader
        Patch with test block
        Show
        Felix Knecht added a comment - Patch for ResourceReader Patch with test block
        Hide
        Felix Knecht added a comment -
        It seems that the problem is situated somewhere deeper. After fixing the ResourceReader I'm getting next errors, depending on the pipeline I use (caching/noncaching).

        The errors I'm getting now (after applying the attached patch for the ResourceReader with the test block):
        Please replace in the test block the file which is read with a huge file you have on your machine (no need to attache a 700MB file here).

        Caching:
        2008-02-14 15:35:20.955:/:INFO: Apache Cocoon Spring Configurator v1.0.2-SNAPSHOT is running in mode 'dev'.
        java.lang.OutOfMemoryError: Java heap space
                at java.util.Arrays.copyOf(Arrays.java:2786)
                at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:71)
                at org.apache.cocoon.servletservice.ServletServiceContext$StatusRetrievableBufferedWrappedResponse$1.write(ServletServiceContext.java:593)
                at java.io.OutputStream.write(OutputStream.java:99)
                at org.apache.cocoon.util.BufferedOutputStream.writeBuffer(BufferedOutputStream.java:137)
                at org.apache.cocoon.util.BufferedOutputStream.realFlush(BufferedOutputStream.java:127)
                at org.apache.cocoon.reading.ResourceReader.processStream(ResourceReader.java:391)
                at org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:420)
                at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processReader(AbstractProcessingPipeline.java:656)
                at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:431)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:72)
                at $Proxy8.process(Unknown Source)
                at org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)
                at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
                at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
                at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
                at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:144)
                at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
                at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
                at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
                at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
                at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
                at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
                at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
                at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
                at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:512)
                at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:479

        Noncaching:
        2008-02-14 15:34:55.179:/:INFO: Apache Cocoon Spring Configurator v1.0.2-SNAPSHOT is running in mode 'dev'.
        java.lang.OutOfMemoryError: Java heap space
                at org.apache.cocoon.util.BufferedOutputStream.incBuffer(BufferedOutputStream.java:148)
                at org.apache.cocoon.util.BufferedOutputStream.write(BufferedOutputStream.java:96)
                at org.apache.cocoon.caching.CachingOutputStream.write(CachingOutputStream.java:69)
                at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
                at org.apache.cocoon.reading.ResourceReader.processStream(ResourceReader.java:386)
                at org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:420)
                at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processReader(AbstractCachingProcessingPipeline.java:866)
                at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:431)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:72)
                at $Proxy8.process(Unknown Source)
                at org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)
                at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
                at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
                at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
                at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:144)
                at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
                at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
                at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
                at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
                at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
                at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
                at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
                at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
                at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:512)
                at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:479)
                at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:230)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        Show
        Felix Knecht added a comment - It seems that the problem is situated somewhere deeper. After fixing the ResourceReader I'm getting next errors, depending on the pipeline I use (caching/noncaching). The errors I'm getting now (after applying the attached patch for the ResourceReader with the test block): Please replace in the test block the file which is read with a huge file you have on your machine (no need to attache a 700MB file here). Caching: 2008-02-14 15:35:20.955:/:INFO: Apache Cocoon Spring Configurator v1.0.2-SNAPSHOT is running in mode 'dev'. java.lang.OutOfMemoryError: Java heap space         at java.util.Arrays.copyOf(Arrays.java:2786)         at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:71)         at org.apache.cocoon.servletservice.ServletServiceContext$StatusRetrievableBufferedWrappedResponse$1.write(ServletServiceContext.java:593)         at java.io.OutputStream.write(OutputStream.java:99)         at org.apache.cocoon.util.BufferedOutputStream.writeBuffer(BufferedOutputStream.java:137)         at org.apache.cocoon.util.BufferedOutputStream.realFlush(BufferedOutputStream.java:127)         at org.apache.cocoon.reading.ResourceReader.processStream(ResourceReader.java:391)         at org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:420)         at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processReader(AbstractProcessingPipeline.java:656)         at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:431)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)         at java.lang.reflect.Method.invoke(Method.java:597)         at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:72)         at $Proxy8.process(Unknown Source)         at org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)         at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)         at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)         at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)         at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:144)         at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)         at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)         at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)         at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)         at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)         at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)         at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)         at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)         at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:512)         at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:479 Noncaching: 2008-02-14 15:34:55.179:/:INFO: Apache Cocoon Spring Configurator v1.0.2-SNAPSHOT is running in mode 'dev'. java.lang.OutOfMemoryError: Java heap space         at org.apache.cocoon.util.BufferedOutputStream.incBuffer(BufferedOutputStream.java:148)         at org.apache.cocoon.util.BufferedOutputStream.write(BufferedOutputStream.java:96)         at org.apache.cocoon.caching.CachingOutputStream.write(CachingOutputStream.java:69)         at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)         at org.apache.cocoon.reading.ResourceReader.processStream(ResourceReader.java:386)         at org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:420)         at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processReader(AbstractCachingProcessingPipeline.java:866)         at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:431)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)         at java.lang.reflect.Method.invoke(Method.java:597)         at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:72)         at $Proxy8.process(Unknown Source)         at org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)         at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)         at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)         at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)         at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:144)         at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)         at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)         at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)         at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)         at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)         at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)         at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)         at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)         at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:512)         at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:479)         at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:230)         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

          People

          • Assignee:
            Jörg Heinicke
            Reporter:
            Felix Knecht
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development