[CXF-2706] AttachmentDeserializer/LazyLoading Attachment Collection enters into continuous while loop for input with missing boundary

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.2, 2.2.3, 2.2.4, 2.2.5, 2.2.6
• Fix Version/s: 2.2.7
• Component/s: Core

Environment

All known platforms to mankind

• Assignee: Daniel Kulp
• Reporter: Mustafa Sezgin
• Created: 11/Mar/10 00:49
• Updated: 23/Mar/10 02:59
• Resolved: 11/Mar/10 17:16

Description

We recently came across an issue in our production environments where we detected http processor threads that had been alive for over a week chewing up CPU. By inspecting thread dumps we found that our external API's (JAX-RS) were the problem with CXF being the culprit. All processor threads had the same stack trace and were all related to POST requests which were multipart based.

Upon further investigation the cause was identified to be an incorrectly sent multipart input with a missing end boundary. The result of this was the LazyAttachmentCollection entering into a continuous loop 'waiting' for more data even though there was none with the client end point having long gone.

I have put together a test case demonstrating this. I have tried to imitate the CXF code path as much as possible (AttachmentInInterceptor).

I consider this to be a fairly serious issue as mistakes like this will likely happen frequently by developers and it would only take 8 of these requests to consume an 8 core cpu and its 'game over man'...

I aim to have a patch implemented for this as soon as possible when i have some free time but im hoping you guys might be able get onto it sooner than me as a fix for this would greatly appreciated...

A sample stack trace from Tomcat is below

                at java.lang.System.arraycopy(Native Method)
                at java.io.PushbackInputStream.unread(PushbackInputStream.java:218)
                at org.apache.cxf.attachment.MimeBodyPartInputStream.hasData(MimeBodyPartInputStream.java:98)
                at org.apache.cxf.attachment.MimeBodyPartInputStream.processBuffer(MimeBodyPartInputStream.java:134)
                at org.apache.cxf.attachment.MimeBodyPartInputStream.read(MimeBodyPartInputStream.java:76)
                at java.io.InputStream.read(InputStream.java:85)
                at org.apache.cxf.attachment.DelegatingInputStream.read(DelegatingInputStream.java:77)
                at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:112)
                at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:75)
                at org.apache.cxf.attachment.AttachmentDataSource.<init>(AttachmentDataSource.java:39)
                at org.apache.cxf.attachment.AttachmentUtil.createAttachment(AttachmentUtil.java:168)
                at org.apache.cxf.attachment.AttachmentDeserializer.createAttachment(AttachmentDeserializer.java:283)
                at org.apache.cxf.attachment.AttachmentDeserializer.readNext(AttachmentDeserializer.java:194)
                at org.apache.cxf.attachment.LazyAttachmentCollection.loadAll(LazyAttachmentCollection.java:52)
                at org.apache.cxf.attachment.LazyAttachmentCollection.size(LazyAttachmentCollection.java:99)
                at org.apache.cxf.jaxrs.ext.MessageContextImpl.createAttachments(MessageContextImpl.java:147)
                at org.apache.cxf.jaxrs.ext.MessageContextImpl.get(MessageContextImpl.java:58)
                at org.apache.cxf.jaxrs.impl.tl.ThreadLocalMessageContext.get(ThreadLocalMessageContext.java:38)
                at org.apache.cxf.jaxrs.utils.multipart.AttachmentUtils.getMultipartBody(AttachmentUtils.java:81)
                at org.apache.cxf.jaxrs.utils.multipart.AttachmentUtils.getAttachments(AttachmentUtils.java:86)
                at org.apache.cxf.jaxrs.provider.MultipartProvider.readFrom(MultipartProvider.java:76)
                at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:827)
                at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameter(JAXRSUtils.java:470)
                at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:435)
                at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:194)
                at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:65)
                at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
                - locked <0x00002aaae3f66ac8> (a org.apache.cxf.phase.PhaseInterceptorChain)
                at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
                at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)
                at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:368)
                at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:146)
                at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:163)
                at org.apache.cxf.transport.servlet.AbstractCXFServlet.doPost(AbstractCXFServlet.java:141)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)