Created attachment 26956 [details]
patch for BodyContentImpl class of trunk

We are reproducing this problem after getting an OutOfMemoryError on BodyContentImpl.reAllocBuff.

java.lang.OutOfMemoryError: Java heap space
        at org.apache.jasper.runtime.BodyContentImpl.reAllocBuff(BodyContentImpl.java:595)
        at org.apache.jasper.runtime.BodyContentImpl.write(BodyContentImpl.java:109)
        at java.io.PrintWriter.write(PrintWriter.java:382)


We are on version 6.0.29, but the cause is still in trunk.

When the buffer needs to increase the size and an OOM error is thrown the bufferSize member variable is increased before the buffer is reallocated. But the BodyContentImpl is accessible for later requests (from the PageContext pool) with a bufferSize greater than the real buffer size, causing the ArrayIndexOutOfBoundsException to be thrown when trying to use the buffer space above the real size.

To solve the problem the bufferSize member variable should be updated only after the buffer array is properly created.

The patch fixes this issue.

Once the JVM issues an OutOfMemoryError (especially for heap), there are no more guarantees that anything works correctly. After such an OOME you must restart the JVM. Relying on things to work just because it seems to is dangerous.

In our case there was no Out-Of-Memory that forced this bug to show. The application still ran fine but the Array-Index out of bounds exception were thrown anyway.

Hmm. Looking at case 2 above, I don't see any way for that to happen unless bufferSize > cb.length. As far as I can see, these are always changed in sync. That leads me to think that there is an OOME that is being missed for some reason.

While I agree 100% with Rainer that once an OOME occurs, all bets are off, I think that having this patch in 7.0.x will at least allow us to determine if the root cause is an OOME or something else.

I have applied a variation of the patch suggested in comment 2 to Tomcat 7.0.x which will be included in 7.0.13 onwards. The patch won't get back-ported to 6.0.x until there is some evidence that it useful. Please test your application with 7.0.13 once released. and let us know how you get on.

If an upgrade to 7.0.13 is not possible, we could provide a binary patch for 6.0.32 *purely for testing purposes*. Let us know if you want to go that route.

No plans for us to upgrade to 7.0 in the near future. We are already testing 6.0.32 with the patch in #2, so no binary patch needed for us, thanks. We will apply Mark patch in trunk to our build to test with the same code.

Although with an OOME we almost always need to restart a JVM in some applications I think this is not always the case with Tomcat (or other server applications) if the lack of memory is due to a server traffic peak and servlets in the applications use many tags with big buffered content. Of course it depends on which point the OOME is thrown.

When this happens the affected request is aborted and all local resources are freed, including huge buffers allocated by tags (which are the cause of the problem for us), so it is not strange that the server can recover in some cases without needing to restart the JVM. 

If this happens (that JVM can free memory after the first OOME to handle following requests) Tomcat is still unusable because of BodyContentImpl instances with an invalid buffer are pooled in PageContext instances at JspFactoryImpl. 

Although the invalid BodyContentImpl instances should be recycled by PageContextImpl.release, in our experience the instances with the invalid buffer are used after the OOME.

Last time we saw this problem, our server seemed to recover from the error after some time (several minutes) throwing the ArrayIndexOutOfBoundsException (AIOOBE) in certain requests very often. After some time working fine (more than 30 minutes) AIOOBE started again in some requests (without any other OOME), we supose because pooled PageContextImpl instances with invalid BodyContextImpl instances were used again needing a big buffer.

So, and always with our experience, we think that in certain circumstances Tomcat can recover from an OOME in BodyContentImp.reAllocBuff, but these pooled invalid BodyContentImpl instances will force to restart JVM to clean them.

Of course, we would like the patch in trunk applied also to 6.0.x ;)

We also used the code in the tomcat repository to patch our current tomcat 6.0.32 and will get back to you after it has been in production for some time. Thanks again for you help!

We have the modified code in production now for about one and half weeks, and the ArrayIndexOutOfBounds exceptions didn't occur anymore.

It seems like the fix works accordingly. We would be glad if this fix could be backported to tomcat 6.0.x

Please leave a comment here, to indicate whether this will be done or whether it won't.

Thank you again for your great help!

Thanks for the feedback. I have proposed the patch for 6.0.x.

Fixed in 6.0.x and will be included in 6.0.33 onwards.

167:                                             &nbsp;</td>
168:                                         <td class="msgtext" align="right">&nbsp;
169:                                             <html:select property="pageIndex" styleClass="simpletext" onchange="javascript:goToSelectedPage();">
170:                                                 <c:forEach var="pages" items="${smsAdminReportForm.pageList}">
171:                                                     <html:option value="${pages.pageIndex}">${pages.pagination}</html:option>
172:                                                 </c:forEach>       
173:                                             </html:select>


Stacktrace:] with root cause
java.lang.OutOfMemoryError: Java heap space
	at org.apache.jasper.runtime.BodyContentImpl.reAllocBuff(BodyContentImpl.java:647)
	at org.apache.jasper.runtime.BodyContentImpl.write(BodyContentImpl.java:145)
	at org.apache.jasper.runtime.BodyContentImpl.write(BodyContentImpl.java:161)
	at org.apache.jsp.smsAdminReportDetails_jsp._jspx_meth_c_005fforEach_005f1(smsAdminReportDetails_jsp.java:1217)
	at org.apache.jsp.smsAdminReportDetails_jsp._jspx_meth_html_005fselect_005f0(smsAdminReportDetails_jsp.java:1179)
	at org.apache.jsp.smsAdminReportDetails_jsp._jspx_meth_c_005fotherwise_005f1(smsAdminReportDetails_jsp.java:1037)
	at org.apache.jsp.smsAdminReportDetails_jsp._jspx_meth_c_005fchoose_005f1(smsAdminReportDetails_jsp.java:931)
	at org.apache.jsp.smsAdminReportDetails_jsp._jspService(smsAdminReportDetails_jsp.java:361)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
	at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1062)
	at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
	at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:386)
	at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:318)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:229)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
	at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)

(In reply to Vijay Ghadage from comment #10)

Please use the users mailing list rather than re-opening a random bug and dumping your stack trace.

Restoring the original parameters...