Uploaded image for project: 'CXF'
  1. CXF
  2. CXF-3180

Jaxrs/Jaxb unmarshaller code regularly causing performance problems & thread lock-up

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.2.6, 2.2.7, 2.2.8, 2.2.9, 2.2.10, 2.3, 2.2.11, 2.2.12, 2.3.1
    • 2.3.2, 2.4
    • None
    • None

    Description

      Please find below the details of the XML parsing performance bottleneck we are seeing when processing REST POST requests with XML bodies.
      The performance issue is caused by the fact that CXF code creates a new JAXB Unmarshaller for every incoming request. The Unmarshaller is created quickly enough but when it is used to do the unmarshalling, it needs to create a Xerces SAXParser, which in turn calls DTDDVFactory.getInstance() and that method is synchronized. When we try to process more than 200-300 requests per second we start seeing a lot of threads blocked in DTDDVFactory.getInstance().

      The thread stack of one of the blocked threads is:

      1390532156@qtp-69728034-390" prio=10 tid=0x00002aab449c0000 nid=0x28f4 waiting for monitor entry [0x000000004da26000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:44)

      waiting to lock <0x00002aaab279ed48> (a java.lang.Class for com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.<init>(XML11Configuration.java:538)
      at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:125)
      at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:86)
      at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at java.lang.Class.newInstance0(Class.java:355)
      at java.lang.Class.newInstance(Class.java:308)
      at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.newInstance(ObjectFactory.java:349)
      at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:154)
      at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:97)
      at com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:102)
      at com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:87)
      at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.<init>(SAXParserImpl.java:332)
      at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:122)
      at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParser(SAXParserFactoryImpl.java:76)
      at javax.xml.bind.helpers.AbstractUnmarshallerImpl.getXMLReader(AbstractUnmarshallerImpl.java:75)
      at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:184)
      at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:191)
      at org.apache.cxf.jaxrs.provider.JAXBElementProvider.unmarshalFromInputStream(JAXBElementProvider.java:216)
      at org.apache.cxf.jaxrs.provider.JAXBElementProvider.doUnmarshal(JAXBElementProvider.java:186)
      at org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:157)
      at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:947)
      at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameter(JAXRSUtils.java:560)
      at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:525)
      at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:230)
      at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:88)
      at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
      locked <0x00002aab1e41a0e8> (a org.apache.cxf.phase.PhaseInterceptorChain)
      at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:109)
      at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:312)
      ...
      The application where the problem is seen defines an XSD file and uses theJAXB compiler to create the data binding java class. It then creates a "RESTResouce" java class that is CXF/JAXRS annotated and contains a method that defines the processing of incoming POST requests. That method has one argument, the JAXB-compiled class, which tells CXF to unmarshall the XML body of the incoming request.

      Just in case, you need more context, here is the top portion of the stack of the thread that is inside the synchronized method:

      "1840040935@qtp-69728034-391" prio=10 tid=0x00002aab44810000 nid=0x2dfa waiting on condition [0x000000005be59000]
      java.lang.Thread.State: RUNNABLE
      at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:211)
      at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:381)
      at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:457)
      at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
      at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:398)
      at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:105)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      at org.eclipse.osgi.internal.loader.BundleLoader.loadClass(BundleLoader.java:326)
      at org.eclipse.osgi.framework.internal.core.BundleHost.loadClass(BundleHost.java:231)
      at org.eclipse.osgi.framework.internal.core.AbstractBundle.loadClass(AbstractBundle.java:1193)
      at org.springframework.osgi.util.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:99)
      at org.springframework.osgi.util.BundleDelegatingClassLoader.loadClass(BundleDelegatingClassLoader.java:156)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      at com.sun.org.apache.xerces.internal.impl.dv.ObjectFactory.findProviderClass(ObjectFactory.java:395)
      at com.sun.org.apache.xerces.internal.impl.dv.ObjectFactory.newInstance(ObjectFactory.java:350)
      at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:59)

      locked <0x00002aaab279ed48> (a java.lang.Class for com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
      at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:44)
      locked <0x00002aaab279ed48> (a java.lang.Class for com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.<init>(XML11Configuration.java:538)
      at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:125)
      at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:86)
      at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
      ...

      Attachments

        Activity

          People

            njiang Willem Jiang
            njiang Willem Jiang
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: