[AXIS2-4424] 100% CPU Utilization during the stress

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.1
• Fix Version/s: 1.6.0
• Component/s: transports
• Environment: Linux, JDK 1.6.0_04-b12, tomcat-6.0.14
• Assignee: Andreas Veithen
• Reporter: Ravi Shanker Reddy
• Created: 09/Jul/09 14:54
• Updated: 07/Apr/10 08:32
• Resolved: 25/Jul/09 18:26

Description

We are continuously sending the SOAP requests to our server, after some time some of the threads will get into following stack trace. The number of threads which falls into the following stack trace will be increase with the time. Once 3 to 4 threads are fot into the below stack trace the server CPU will reaces to 100%. We debugged the issue and find that the problem is with the below method

org.apache.axiom.soap.impl.builder StAXSOAPModelBuilder
Method:
public SOAPEnvelope getSOAPEnvelope() throws OMException {
while ((envelope == null) && !done)

{ next(); }

return envelope;
}

In some specific conditions, the while loop never ends which resulted into 100% CPU.

We enabled the TRACING and found that, once any thread falls in the this loop getting following statement in log file continuously.

Please note that, there is not issue with SOAP request which we are sending, we tested the SOAP requests individually and all are getting proper response.

09 Jul 2009 09:19:10 TRACE [http-8080-17] builder.StAXOMBuilder - START_DOCUMENT:
09 Jul 2009 09:19:10 TRACE [http-8080-17] builder.StAXOMBuilder - START_DOCUMENT:
09 Jul 2009 09:19:10 TRACE [http-8080-17] builder.StAXOMBuilder - START_DOCUMENT:
09 Jul 2009 09:19:10 TRACE [http-8080-17] builder.StAXOMBuilder - START_DOCUMENT:
09 Jul 2009 09:19:10 TRACE [http-8080-17] builder.StAXOMBuilder - START_DOCUMENT:
09 Jul 2009 09:19:10 TRACE [http-8080-17] builder.StAXOMBuilder - START_DOCUMENT:

Stack Trace of The Thread:-
http-8080-34" daemon prio=10 tid=0x7addb400 nid=0x2423 runnable [0x794ee000..0x794ef030]
java.lang.Thread.State: RUNNABLE
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:177)
at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.getSOAPEnvelope(StAXSOAPModelBuilder.java:156)
at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.<init>(StAXSOAPModelBuilder.java:105)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:53)
at org.apache.axis2.transport.TransportUtils.createDocumentElement(TransportUtils.java:164)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:112)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:270)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:133)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.valves.RequestDumperValve.invoke(RequestDumperValve.java:151)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)

Issue Links

• relates to: AXIOM-74 StAXUtils incorrectly assumes that XMLInputFactory and XMLOutputFactory instances are thread safe