Camel
  1. Camel
  2. CAMEL-3776

Improve the performance for JAXB data format

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.5, 2.9.2, 2.10.0
    • Component/s: camel-jaxb
    • Labels:
      None

      Description

      We should use the ServicePool in camel-core to pool JAXB / marshaller / unmarshaller.
      Of course ppl should be able to customize pool settings and if to be used.

      Also mind that for type converters with JAXB you do not configure a JAXB data format. And thus we should
      still be able to configure and use pooling, so the type converter can be more efficient.

      1. jaxb-pool.patch
        21 kB
        Frank Kootte

        Activity

        Hide
        Frank Kootte added a comment -

        As I've ran into performance issues parsing a lot of smaller xml payloads using the camel-jaxb component I am interested to hear if any work has been done with regards to the subject.

        Show
        Frank Kootte added a comment - As I've ran into performance issues parsing a lot of smaller xml payloads using the camel-jaxb component I am interested to hear if any work has been done with regards to the subject.
        Hide
        Christian Müller added a comment -

        Hey Frank,
        nobody is working on it because it's target for Camel 3.0.0. At present we are working on Camel 2.10.0.
        If you would like to see this fixed earlier, feel free to vote for it (you has to be logged in into JIRA to be able to vote). I think we could prepone it if needed.
        Best,
        Christian

        Show
        Christian Müller added a comment - Hey Frank, nobody is working on it because it's target for Camel 3.0.0. At present we are working on Camel 2.10.0. If you would like to see this fixed earlier, feel free to vote for it (you has to be logged in into JIRA to be able to vote). I think we could prepone it if needed. Best, Christian
        Hide
        Frank Kootte added a comment - - edited

        Thank you for the swift, quick and clear reply.

        Meanwhile - at TomTom - I have patched the component myself as we've been suffering from BLOCKED threads while retrieving instances of Marshallers & Unmarshallers ( see below stacktrace ) under heavy load. I've added a very basic implementation surrounding an Apache Commons Pool pooling - JAXBContexts ( not strictly required ), Marshallers and Unmarshallers. I will add the patch to the ticket as an attachment when it indeed successfully endures a high load test.

        java.lang.Thread.State: BLOCKED
            at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:44)
            - waiting to lock <6e5a3923> (a java.lang.Class) owned by "http-8080-17" t@158
            at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.<init>(XML11Configuration.java:538)
            at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:125)
            at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:86)
            at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
            at java.lang.Class.newInstance0(Class.java:355)
            at java.lang.Class.newInstance(Class.java:308)
            at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.newInstance(ObjectFactory.java:349)
            at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:154)
            at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:97)
            at com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:102)
            at com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:87)
            at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.<init>(SAXParserImpl.java:332)
            at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:122)
            at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParser(SAXParserFactoryImpl.java:76)
            at javax.xml.bind.helpers.AbstractUnmarshallerImpl.getXMLReader(AbstractUnmarshallerImpl.java:86)
            at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:137)
            at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:184)
            at org.apache.camel.converter.jaxb.FallbackTypeConverter.unmarshal(FallbackTypeConverter.java:199)
        
        Show
        Frank Kootte added a comment - - edited Thank you for the swift, quick and clear reply. Meanwhile - at TomTom - I have patched the component myself as we've been suffering from BLOCKED threads while retrieving instances of Marshallers & Unmarshallers ( see below stacktrace ) under heavy load. I've added a very basic implementation surrounding an Apache Commons Pool pooling - JAXBContexts ( not strictly required ), Marshallers and Unmarshallers. I will add the patch to the ticket as an attachment when it indeed successfully endures a high load test. java.lang. Thread .State: BLOCKED at com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:44) - waiting to lock <6e5a3923> (a java.lang. Class ) owned by "http-8080-17" t@158 at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.<init>(XML11Configuration.java:538) at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:125) at com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:86) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at java.lang. Class .newInstance0( Class .java:355) at java.lang. Class .newInstance( Class .java:308) at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.newInstance(ObjectFactory.java:349) at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:154) at com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:97) at com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:102) at com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:87) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.<init>(SAXParserImpl.java:332) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:122) at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParser(SAXParserFactoryImpl.java:76) at javax.xml.bind.helpers.AbstractUnmarshallerImpl.getXMLReader(AbstractUnmarshallerImpl.java:86) at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:137) at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:184) at org.apache.camel.converter.jaxb.FallbackTypeConverter.unmarshal(FallbackTypeConverter.java:199)
        Hide
        Frank Kootte added a comment - - edited

        I've attached the patch in its 1st incarnation. I would appreciate the feedback required to land an ( amended ) version in the Camel trunk. We are in the process of testing the patch in our performance test environment. I will attach the findings as soon as they are available.

        Show
        Frank Kootte added a comment - - edited I've attached the patch in its 1st incarnation. I would appreciate the feedback required to land an ( amended ) version in the Camel trunk. We are in the process of testing the patch in our performance test environment. I will attach the findings as soon as they are available.
        Hide
        Daniel Kulp added a comment - - edited

        You may not need to pool the unmarshallers. If you use the unmarshall method that takes the XMLStreamReader, then it doesn't need to create a parser and the entire issue is bypassed. (this is what CXF does btw) Thus, you may be able to call convertTo(XMLStreamReader.class) and pass that in.

        Show
        Daniel Kulp added a comment - - edited You may not need to pool the unmarshallers. If you use the unmarshall method that takes the XMLStreamReader, then it doesn't need to create a parser and the entire issue is bypassed. (this is what CXF does btw) Thus, you may be able to call convertTo(XMLStreamReader.class) and pass that in.
        Hide
        Christian Müller added a comment -

        I will have a look at it later or tomorrow...

        Show
        Christian Müller added a comment - I will have a look at it later or tomorrow...
        Hide
        Christian Müller added a comment -

        It doesn't work out of the box. The route

        from("direct:start")
          .convertBodyTo(XMLStreamReader.class)
          .unmarshal(jaxb)
          .to("mock:result");
        

        throws

        Exception in thread "pool-1-thread-2" org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: <purchaseOrder name='Wine' amount='123.45' price='2.22'/>]
        	at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1237)
        	at org.apache.camel.util.ExchangeHelper.extractResultBody(ExchangeHelper.java:509)
        	at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:442)
        	at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:438)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:126)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:342)
        	at org.apache.camel.example.UnmarshalTest$1.run(UnmarshalTest.java:73)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        	at java.lang.Thread.run(Thread.java:680)
        Caused by: org.apache.camel.InvalidPayloadException: No body available of type: javax.xml.stream.XMLStreamReader but has value: <purchaseOrder name='Wine' amount='123.45' price='2.22'/> of type: java.lang.String on: Message: <purchaseOrder name='Wine' amount='123.45' price='2.22'/>. Caused by: No type converter available to convert from type: java.lang.String to the required type: javax.xml.stream.XMLStreamReader with value <purchaseOrder name='Wine' amount='123.45' price='2.22'/>. Exchange[Message: <purchaseOrder name='Wine' amount='123.45' price='2.22'/>]. Caused by: [org.apache.camel.NoTypeConversionAvailableException - No type converter available to convert from type: java.lang.String to the required type: javax.xml.stream.XMLStreamReader with value <purchaseOrder name='Wine' amount='123.45' price='2.22'/>]
        	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:102)
        	at org.apache.camel.processor.ConvertBodyProcessor.process(ConvertBodyProcessor.java:58)
        	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333)
        	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223)
        	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
        	at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122)
        	at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61)
        	at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
        	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)
        	at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:63)
        	at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:352)
        	at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:324)
        	at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:223)
        	at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:324)
        	at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:169)
        	at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:111)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:124)
        	... 5 more
        Caused by: org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: java.lang.String to the required type: javax.xml.stream.XMLStreamReader with value <purchaseOrder name='Wine' amount='123.45' price='2.22'/>
        	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:147)
        	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:100)
        	... 43 more
        

        If I change the route to:

        from("direct:start")
          .convertBodyTo(InputStream.class)
          .convertBodyTo(XMLStreamReader.class)
          .unmarshal(jaxb)
          .to("mock:result");
        

        I get the following exception:

        Exception in thread "pool-1-thread-14" org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: com.ctc.wstx.sr.ValidatingStreamReader@4987b287]
        	at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1237)
        	at org.apache.camel.util.ExchangeHelper.extractResultBody(ExchangeHelper.java:509)
        	at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:442)
        	at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:438)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:126)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:342)
        	at org.apache.camel.example.UnmarshalTest$1.run(UnmarshalTest.java:73)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        	at java.lang.Thread.run(Thread.java:680)
        Caused by: org.apache.camel.InvalidPayloadException: No body available of type: java.io.InputStream but has value: com.ctc.wstx.sr.ValidatingStreamReader@4987b287 of type: com.ctc.wstx.sr.ValidatingStreamReader on: Message: com.ctc.wstx.sr.ValidatingStreamReader@4987b287. Caused by: No type converter available to convert from type: com.ctc.wstx.sr.ValidatingStreamReader to the required type: java.io.InputStream with value com.ctc.wstx.sr.ValidatingStreamReader@4987b287. Exchange[Message: com.ctc.wstx.sr.ValidatingStreamReader@4987b287]. Caused by: [org.apache.camel.NoTypeConversionAvailableException - No type converter available to convert from type: com.ctc.wstx.sr.ValidatingStreamReader to the required type: java.io.InputStream with value com.ctc.wstx.sr.ValidatingStreamReader@4987b287]
        	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:102)
        	at org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:118)
        	at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:52)
        	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333)
        	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223)
        	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
        	at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122)
        	at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61)
        	at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
        	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)
        	at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:63)
        	at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:352)
        	at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:324)
        	at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:223)
        	at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:324)
        	at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:169)
        	at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:111)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:124)
        	... 5 more
        Caused by: org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: com.ctc.wstx.sr.ValidatingStreamReader to the required type: java.io.InputStream with value com.ctc.wstx.sr.ValidatingStreamReader@4987b287
        	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:147)
        	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:100)
        	... 44 more
        
        Show
        Christian Müller added a comment - It doesn't work out of the box. The route from("direct:start") .convertBodyTo(XMLStreamReader.class) .unmarshal(jaxb) .to("mock:result"); throws Exception in thread "pool-1-thread-2" org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: <purchaseOrder name='Wine' amount='123.45' price='2.22'/>] at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1237) at org.apache.camel.util.ExchangeHelper.extractResultBody(ExchangeHelper.java:509) at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:442) at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:438) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:126) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:342) at org.apache.camel.example.UnmarshalTest$1.run(UnmarshalTest.java:73) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: org.apache.camel.InvalidPayloadException: No body available of type: javax.xml.stream.XMLStreamReader but has value: <purchaseOrder name='Wine' amount='123.45' price='2.22'/> of type: java.lang.String on: Message: <purchaseOrder name='Wine' amount='123.45' price='2.22'/>. Caused by: No type converter available to convert from type: java.lang.String to the required type: javax.xml.stream.XMLStreamReader with value <purchaseOrder name='Wine' amount='123.45' price='2.22'/>. Exchange[Message: <purchaseOrder name='Wine' amount='123.45' price='2.22'/>]. Caused by: [org.apache.camel.NoTypeConversionAvailableException - No type converter available to convert from type: java.lang.String to the required type: javax.xml.stream.XMLStreamReader with value <purchaseOrder name='Wine' amount='123.45' price='2.22'/>] at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:102) at org.apache.camel.processor.ConvertBodyProcessor.process(ConvertBodyProcessor.java:58) at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223) at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.Pipeline.process(Pipeline.java:117) at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122) at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61) at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86) at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:63) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:352) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:324) at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:223) at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:324) at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:169) at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:111) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:124) ... 5 more Caused by: org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: java.lang.String to the required type: javax.xml.stream.XMLStreamReader with value <purchaseOrder name='Wine' amount='123.45' price='2.22'/> at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:147) at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:100) ... 43 more If I change the route to: from("direct:start") .convertBodyTo(InputStream.class) .convertBodyTo(XMLStreamReader.class) .unmarshal(jaxb) .to("mock:result"); I get the following exception: Exception in thread "pool-1-thread-14" org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: com.ctc.wstx.sr.ValidatingStreamReader@4987b287] at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1237) at org.apache.camel.util.ExchangeHelper.extractResultBody(ExchangeHelper.java:509) at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:442) at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:438) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:126) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:342) at org.apache.camel.example.UnmarshalTest$1.run(UnmarshalTest.java:73) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: org.apache.camel.InvalidPayloadException: No body available of type: java.io.InputStream but has value: com.ctc.wstx.sr.ValidatingStreamReader@4987b287 of type: com.ctc.wstx.sr.ValidatingStreamReader on: Message: com.ctc.wstx.sr.ValidatingStreamReader@4987b287. Caused by: No type converter available to convert from type: com.ctc.wstx.sr.ValidatingStreamReader to the required type: java.io.InputStream with value com.ctc.wstx.sr.ValidatingStreamReader@4987b287. Exchange[Message: com.ctc.wstx.sr.ValidatingStreamReader@4987b287]. Caused by: [org.apache.camel.NoTypeConversionAvailableException - No type converter available to convert from type: com.ctc.wstx.sr.ValidatingStreamReader to the required type: java.io.InputStream with value com.ctc.wstx.sr.ValidatingStreamReader@4987b287] at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:102) at org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:118) at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:52) at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223) at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.Pipeline.process(Pipeline.java:117) at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122) at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61) at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86) at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:63) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:352) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:324) at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:223) at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:324) at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:169) at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:111) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:124) ... 5 more Caused by: org.apache.camel.NoTypeConversionAvailableException: No type converter available to convert from type: com.ctc.wstx.sr.ValidatingStreamReader to the required type: java.io.InputStream with value com.ctc.wstx.sr.ValidatingStreamReader@4987b287 at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:147) at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:100) ... 44 more
        Hide
        Frank Kootte added a comment - - edited

        Peculiar. The unit tests all succeed for me. I have run the ones in the camel-jaxb only in all honesty. Our routes invoking the FallbackTypeConverter do no trigger the exception either. Monday I will look into the issue. Regarding the way cxf deals with avoiding the creation of an SAXParser I personally feel that dealing with all occasions equally an in uniform way when no obvious negative effects are introduced seems to be a bit cleaner.

        Show
        Frank Kootte added a comment - - edited Peculiar. The unit tests all succeed for me. I have run the ones in the camel-jaxb only in all honesty. Our routes invoking the FallbackTypeConverter do no trigger the exception either. Monday I will look into the issue. Regarding the way cxf deals with avoiding the creation of an SAXParser I personally feel that dealing with all occasions equally an in uniform way when no obvious negative effects are introduced seems to be a bit cleaner.
        Hide
        Christian Müller added a comment -

        Frank, I tried whether Camel can deal with the XMLStreamReader conversion out of the box (without your patch). So, there is nothing wrong with it.
        I made a couple of stress tests. I unmarshaled the XML

        <purchaseOrder name='Wine' amount='123.45' price='2.22'/>
        

        10000 times with

        ExecutorService pool = Executors.newFixedThreadPool(20);
        long start = System.currentTimeMillis();
        for (int i = 0; i < counter; i++) {
            pool.execute(new Runnable() {
                public void run() {
                    template.sendBody(payload);
                    latch.countDown();
                }
            });
        }
        

        The numbers I got for different scenarios are:

        • with the current implementation it took 6440ms
        • with using a GenericObjectPool from commons-pool (maxActive = 20) it took 3323ms
        • with using a ReentrantLock in JaxbDataFormat.unmarshal() for the lines where the Unmarshaller is used, it took 2813ms

        I didn't had the time to look what's necessary to get the convertBodyTo(XMLStreamReader.class) working an what the performance is.

        May be the ReentrantLock is a smart solution without the "complexity" of pooling objects?

        Show
        Christian Müller added a comment - Frank, I tried whether Camel can deal with the XMLStreamReader conversion out of the box (without your patch). So, there is nothing wrong with it. I made a couple of stress tests. I unmarshaled the XML <purchaseOrder name='Wine' amount='123.45' price='2.22'/> 10000 times with ExecutorService pool = Executors.newFixedThreadPool(20); long start = System .currentTimeMillis(); for ( int i = 0; i < counter; i++) { pool.execute( new Runnable () { public void run() { template.sendBody(payload); latch.countDown(); } }); } The numbers I got for different scenarios are: with the current implementation it took 6440ms with using a GenericObjectPool from commons-pool (maxActive = 20) it took 3323ms with using a ReentrantLock in JaxbDataFormat.unmarshal() for the lines where the Unmarshaller is used, it took 2813ms I didn't had the time to look what's necessary to get the convertBodyTo(XMLStreamReader.class) working an what the performance is. May be the ReentrantLock is a smart solution without the "complexity" of pooling objects?
        Hide
        Christian Müller added a comment - - edited

        I did some tests with the XMLStreamReader in JaxbDataFormat, but without luck. I get NullPointerException from JAXB (see below). I did the following changes:
        In the JaxbDataFormat.doStart() method I create an Unmarshaller and set this to an new instance variable.
        In JaxbDataFormat.unmarshal() I converted the InputStrem into an XMLStreamReader:

        XMLStreamReader xmlReader = exchange.getContext().getTypeConverter().convertTo(XMLStreamReader.class, exchange, stream);
        

        and pass this reader into the Unmarshaller's unmarshal method.

        Exception in thread "pool-2-thread-1" org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: <purchaseOrder name='Wine' amount='123.45' price='2.22'/>]
        	at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1237)
        	at org.apache.camel.util.ExchangeHelper.extractResultBody(ExchangeHelper.java:509)
        	at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:442)
        	at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:438)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:126)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:342)
        	at org.apache.camel.example.DataFormatConcurrentTest$1.run(DataFormatConcurrentTest.java:49)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        	at java.lang.Thread.run(Thread.java:680)
        Caused by: java.lang.NullPointerException
        	at com.sun.xml.bind.v2.runtime.Coordinator.popCoordinator(Coordinator.java:157)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:461)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:242)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:176)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:360)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:332)
        	at org.apache.camel.converter.jaxb.JaxbDataFormat.unmarshal(JaxbDataFormat.java:144)
        	at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:58)
        	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333)
        	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223)
        	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
        	at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122)
        	at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61)
        	at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
        	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)
        	at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:63)
        	at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:352)
        	at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:324)
        	at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:223)
        	at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:324)
        	at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:169)
        	at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:111)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:124)
        	... 5 more
        
        Show
        Christian Müller added a comment - - edited I did some tests with the XMLStreamReader in JaxbDataFormat, but without luck. I get NullPointerException from JAXB (see below). I did the following changes: In the JaxbDataFormat.doStart() method I create an Unmarshaller and set this to an new instance variable. In JaxbDataFormat.unmarshal() I converted the InputStrem into an XMLStreamReader: XMLStreamReader xmlReader = exchange.getContext().getTypeConverter().convertTo(XMLStreamReader.class, exchange, stream); and pass this reader into the Unmarshaller's unmarshal method. Exception in thread "pool-2-thread-1" org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: <purchaseOrder name='Wine' amount='123.45' price='2.22'/>] at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1237) at org.apache.camel.util.ExchangeHelper.extractResultBody(ExchangeHelper.java:509) at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:442) at org.apache.camel.impl.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:438) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:126) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:342) at org.apache.camel.example.DataFormatConcurrentTest$1.run(DataFormatConcurrentTest.java:49) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: java.lang.NullPointerException at com.sun.xml.bind.v2.runtime.Coordinator.popCoordinator(Coordinator.java:157) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:461) at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:242) at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:176) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:360) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:332) at org.apache.camel.converter.jaxb.JaxbDataFormat.unmarshal(JaxbDataFormat.java:144) at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:58) at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223) at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.Pipeline.process(Pipeline.java:117) at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122) at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61) at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86) at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:63) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:352) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:324) at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:223) at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:324) at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:169) at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:111) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:124) ... 5 more
        Hide
        Christian Müller added a comment -

        I committed r1308165 which improves the performance significant. I unmarshaled the XML I mentioned before 10000 times:
        before: 7042ms
        after: 2884ms

        However, I only create an Unmarshaller instance variable at start and with the help of an ReentrantLock I make sure it's only used one a time. I will let this issue open until others agree this is the final solution or bring up a better/faster one.

        Show
        Christian Müller added a comment - I committed r1308165 which improves the performance significant. I unmarshaled the XML I mentioned before 10000 times: before: 7042ms after: 2884ms However, I only create an Unmarshaller instance variable at start and with the help of an ReentrantLock I make sure it's only used one a time. I will let this issue open until others agree this is the final solution or bring up a better/faster one.
        Hide
        Frank Kootte added a comment -

        Great work! I am not particularly bound to the pooled implementation to be honest. Let me fetch your code put it under high load ( 300 requests per second ) and see if we encounter the synchronization issues as the stacktrace above indicates - given the nature of the fix I presume this should work out. The BLOCKED threads only occurred when we reached about 150 requests per second with a lot of identical payloads - I will keep you posted on the results ( expect them today ).

        Show
        Frank Kootte added a comment - Great work! I am not particularly bound to the pooled implementation to be honest. Let me fetch your code put it under high load ( 300 requests per second ) and see if we encounter the synchronization issues as the stacktrace above indicates - given the nature of the fix I presume this should work out. The BLOCKED threads only occurred when we reached about 150 requests per second with a lot of identical payloads - I will keep you posted on the results ( expect them today ).
        Hide
        Frank Kootte added a comment - - edited

        The change I made was against tag/2.9.1 and I see you patched trunk ( makes more sense ). Question for us is at which request rate congestion will occur around the ReentrantLock compared to a pooled solution - which can scale on demand which is of our interest as we're dealing with quite the amount of requests per second on peak load. Load tests on the pooled impl I did provide a rough 300+ requests per second improvement showing no congestion until we consume 100% CPU resources. I am in the process of deploying a build of 2.10-SNAPSHOT to perform the exact same test to see if and how the behavior between the 2 solutions differ when stressed.

        Show
        Frank Kootte added a comment - - edited The change I made was against tag/2.9.1 and I see you patched trunk ( makes more sense ). Question for us is at which request rate congestion will occur around the ReentrantLock compared to a pooled solution - which can scale on demand which is of our interest as we're dealing with quite the amount of requests per second on peak load. Load tests on the pooled impl I did provide a rough 300+ requests per second improvement showing no congestion until we consume 100% CPU resources. I am in the process of deploying a build of 2.10-SNAPSHOT to perform the exact same test to see if and how the behavior between the 2 solutions differ when stressed.
        Hide
        Frank Kootte added a comment - - edited

        Our code predominantly ( actually only ) uses the FallbackTypeConverter ( which I originally patched ). Applying a patch with the same semantics as the patch you applied to the FallbackTypeConverter would solve the issue for all ways to apply JAXB conversions. A ThreadLocal instance containing a per Thread Unmarshaller would be the cheapest solution ( in this case I am comparing it to pooling ) I figure - it would however retain the memory until ( if! ) the spawned Threads would be removed from the application server ThreadPool releasing their resources and making them available for GC - which would be more controllable given the Pool usage. Given the nature of the class - used for multiple JAXBContext's and therefore requiring multiple Unmarshallers for different types this seems an unlikely candidate to be achieved in a fashionable manner. What is your vision upon the subject exactly ? Perhaps - while we are at it - we should also look into removing the duplication we see in that region and have a single solution for both callsites.

        Show
        Frank Kootte added a comment - - edited Our code predominantly ( actually only ) uses the FallbackTypeConverter ( which I originally patched ). Applying a patch with the same semantics as the patch you applied to the FallbackTypeConverter would solve the issue for all ways to apply JAXB conversions. A ThreadLocal instance containing a per Thread Unmarshaller would be the cheapest solution ( in this case I am comparing it to pooling ) I figure - it would however retain the memory until ( if! ) the spawned Threads would be removed from the application server ThreadPool releasing their resources and making them available for GC - which would be more controllable given the Pool usage. Given the nature of the class - used for multiple JAXBContext's and therefore requiring multiple Unmarshallers for different types this seems an unlikely candidate to be achieved in a fashionable manner. What is your vision upon the subject exactly ? Perhaps - while we are at it - we should also look into removing the duplication we see in that region and have a single solution for both callsites.
        Hide
        Christian Müller added a comment -

        Hello Frank!

        If we agree about the solution, I will back port it of course to the maintained camel-2.9.x and camel-2.8.x branch.
        I highly interested which of the both solutions works better (faster) for you. If it's the pooled solution, I don't have anything against to this solution. I always try to start with the simplest solution which full fill the requirements...
        I also try to get a better understanding what Dan suggested. But at present, I only got some NullPointerExceptions from JAXB when I converted the InputStream into a XMLStreamReader and pass this to the Unmarshaller... Keep you posted...

        I see the issue with the FallbackTypeConverter. I will fix this also. But my recommendation is to use the JAXBDataFormat explicitly. Imagine you have a route where you use the FallbackTypeConverter 10 times for the same JAXBContext. All this 10 routes will use the same Unmarshaller which will be less per formant (more locking), than to explicitly use 10 different JAXBDataFormat instances (of course with the same JAXBContext). This will require less locking because the Unmarshaller is shared per JAXBDataFormat and not per JAXBContext. Make sense for you?

        I we go for the pooled solution (if it's faster), we should use/register the pool capabilities Camel provides. At present, I'm not familiar enough with this to suggest a solution. Will wait for the performance results of your tests and if I'm able to get the XMLStreamReader stuff working...

        Best,
        Christian

        Show
        Christian Müller added a comment - Hello Frank! If we agree about the solution, I will back port it of course to the maintained camel-2.9.x and camel-2.8.x branch. I highly interested which of the both solutions works better (faster) for you. If it's the pooled solution, I don't have anything against to this solution. I always try to start with the simplest solution which full fill the requirements... I also try to get a better understanding what Dan suggested. But at present, I only got some NullPointerExceptions from JAXB when I converted the InputStream into a XMLStreamReader and pass this to the Unmarshaller... Keep you posted... I see the issue with the FallbackTypeConverter. I will fix this also. But my recommendation is to use the JAXBDataFormat explicitly. Imagine you have a route where you use the FallbackTypeConverter 10 times for the same JAXBContext. All this 10 routes will use the same Unmarshaller which will be less per formant (more locking), than to explicitly use 10 different JAXBDataFormat instances (of course with the same JAXBContext). This will require less locking because the Unmarshaller is shared per JAXBDataFormat and not per JAXBContext. Make sense for you? I we go for the pooled solution (if it's faster), we should use/register the pool capabilities Camel provides. At present, I'm not familiar enough with this to suggest a solution. Will wait for the performance results of your tests and if I'm able to get the XMLStreamReader stuff working... Best, Christian
        Hide
        Frank Kootte added a comment -

        Hye Christian - appreciate the fenomenal collaboration !

        Indeed would the solution induce more locking giving the current approach. This is actually the reasoning behind my initial solution - pooling contexts, marshallers and unmarshallers - which in hindsight is somewhat overkill for common usecases - not everbody needs a highly scalable solution catering for high load as we are currently dealing with. The automagically expanding and shrinking pool would load on demand and scale to extremely high load. As we have a lot of routes using the FallbackTypeConverter I will try to amend a single end-to-end flow to use the JaxbDataFormat so we can compare how the different approaches when stressed. I hope I will be able to provide you feedback on performance tomorrow but given the work required I wont make promises anymore I am not sure I can live up to.

        With regards to the pooling solution camel provides - I briefly looked into it before I decided to go for the Apache Commons Pool approach - I think it is worth considering to refactor the camel pool to use the APC as I think it contributes to familiarity and mantainability to reuse the proven solution instead of a special camel catered crafted approach offering no directly obvious benefits.

        Hopefully I will be able to provide you with some conclusive numbers soon and we can finalise this patch.

        Cheers

        Show
        Frank Kootte added a comment - Hye Christian - appreciate the fenomenal collaboration ! Indeed would the solution induce more locking giving the current approach. This is actually the reasoning behind my initial solution - pooling contexts, marshallers and unmarshallers - which in hindsight is somewhat overkill for common usecases - not everbody needs a highly scalable solution catering for high load as we are currently dealing with. The automagically expanding and shrinking pool would load on demand and scale to extremely high load. As we have a lot of routes using the FallbackTypeConverter I will try to amend a single end-to-end flow to use the JaxbDataFormat so we can compare how the different approaches when stressed. I hope I will be able to provide you feedback on performance tomorrow but given the work required I wont make promises anymore I am not sure I can live up to. With regards to the pooling solution camel provides - I briefly looked into it before I decided to go for the Apache Commons Pool approach - I think it is worth considering to refactor the camel pool to use the APC as I think it contributes to familiarity and mantainability to reuse the proven solution instead of a special camel catered crafted approach offering no directly obvious benefits. Hopefully I will be able to provide you with some conclusive numbers soon and we can finalise this patch. Cheers
        Hide
        Christian Müller added a comment -

        I rerun the test with the following route

        from("direct:unmarshalFallback")
          .convertBodyTo(PurchaseOrder.class)
          .to("mock:result");
        

        and it still took 6921ms (with my latest patch). So, I have to work further on this...

        Show
        Christian Müller added a comment - I rerun the test with the following route from( "direct:unmarshalFallback" ) .convertBodyTo(PurchaseOrder.class) .to( "mock:result" ); and it still took 6921ms (with my latest patch). So, I have to work further on this...
        Hide
        Christian Müller added a comment - - edited

        Committed r1308524
        By caching the Unmarshaller in FallbackTypeConverter like the JAXBContext, I could reduce it to 3328ms

        Show
        Christian Müller added a comment - - edited Committed r1308524 By caching the Unmarshaller in FallbackTypeConverter like the JAXBContext, I could reduce it to 3328ms
        Hide
        Christian Müller added a comment -

        By using the XMLStreamReader in FallbackTypeConverter and a shared Unmarshaller, I also got the NullpointerException from JAXB:

        java.lang.NullPointerException
        	at com.sun.xml.bind.v2.runtime.Coordinator.popCoordinator(Coordinator.java:157)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:461)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:242)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:176)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:360)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:332)
        	at org.apache.camel.converter.jaxb.FallbackTypeConverter.unmarshal(FallbackTypeConverter.java:204)
        	at org.apache.camel.converter.jaxb.FallbackTypeConverter.unmarshall(FallbackTypeConverter.java:139)
        	at org.apache.camel.converter.jaxb.FallbackTypeConverter.convertTo(FallbackTypeConverter.java:91)
        	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:204)
        	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:141)
        	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:100)
        	at org.apache.camel.processor.ConvertBodyProcessor.process(ConvertBodyProcessor.java:58)
        	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333)
        	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223)
        	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)
        	at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        	at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122)
        	at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
        	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
        	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61)
        	at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
        	at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
        	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
        	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)
        	at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:63)
        	at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:352)
        	at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:324)
        	at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:223)
        	at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:324)
        	at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:169)
        	at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:111)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:124)
        	at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:342)
        	at org.apache.camel.example.DataFormatConcurrentTest$2.run(DataFormatConcurrentTest.java:74)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        	at java.lang.Thread.run(Thread.java:680)
        

        And as documented in [1], the Unmarshaller from Jaxb Impl RI (Oracle) is not thread safe.
        I pinged Dan to get a pointer to the CXF code where they share the Unmarshaller...

        [1] http://jaxb.java.net/guide/Performance_and_thread_safety.html

        Show
        Christian Müller added a comment - By using the XMLStreamReader in FallbackTypeConverter and a shared Unmarshaller, I also got the NullpointerException from JAXB: java.lang.NullPointerException at com.sun.xml.bind.v2.runtime.Coordinator.popCoordinator(Coordinator.java:157) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:461) at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.handleStartElement(StAXStreamConnector.java:242) at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:176) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:360) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:332) at org.apache.camel.converter.jaxb.FallbackTypeConverter.unmarshal(FallbackTypeConverter.java:204) at org.apache.camel.converter.jaxb.FallbackTypeConverter.unmarshall(FallbackTypeConverter.java:139) at org.apache.camel.converter.jaxb.FallbackTypeConverter.convertTo(FallbackTypeConverter.java:91) at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:204) at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:141) at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:100) at org.apache.camel.processor.ConvertBodyProcessor.process(ConvertBodyProcessor.java:58) at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:333) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:223) at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.Pipeline.process(Pipeline.java:117) at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:122) at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:61) at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150) at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117) at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86) at org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:63) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:352) at org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:324) at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:223) at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:324) at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:169) at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:111) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:124) at org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:342) at org.apache.camel.example.DataFormatConcurrentTest$2.run(DataFormatConcurrentTest.java:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) And as documented in [1] , the Unmarshaller from Jaxb Impl RI (Oracle) is not thread safe. I pinged Dan to get a pointer to the CXF code where they share the Unmarshaller... [1] http://jaxb.java.net/guide/Performance_and_thread_safety.html
        Hide
        Christian Müller added a comment -

        Committed r1308593
        Improved FallbackTypeConverter.marshall() a bit. Only set Marshaller properties if they differ from the default one (because they are expensive)
        before: took 3214ms
        after: took 2680ms

        Show
        Christian Müller added a comment - Committed r1308593 Improved FallbackTypeConverter.marshall() a bit. Only set Marshaller properties if they differ from the default one (because they are expensive) before: took 3214ms after: took 2680ms
        Hide
        Frank Kootte added a comment - - edited

        I am patching our 2.9.1 branch with you latest commits and will execute performance tests against them so we can compare those against the results using the pooled impl I did.
        Unit tests fail after backporting.

        Caused by: java.util.ConcurrentModificationException
        	at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
        	at java.util.AbstractList$Itr.next(AbstractList.java:343)
        	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:203)
        	at org.apache.camel.impl.converter.LazyLoadingTypeConverter.doConvertTo(LazyLoadingTypeConverter.java:47)
        	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:141)
        	... 42 more
        
        

        Attempting to backport BaseTypeConverterRegistry as well to solve the issue.

        Show
        Frank Kootte added a comment - - edited I am patching our 2.9.1 branch with you latest commits and will execute performance tests against them so we can compare those against the results using the pooled impl I did. Unit tests fail after backporting. Caused by: java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) at java.util.AbstractList$Itr.next(AbstractList.java:343) at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:203) at org.apache.camel.impl.converter.LazyLoadingTypeConverter.doConvertTo(LazyLoadingTypeConverter.java:47) at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:141) ... 42 more Attempting to backport BaseTypeConverterRegistry as well to solve the issue.
        Hide
        Frank Kootte added a comment -

        I did an update on TRUNK and tried to run the unit tests on camel-jaxb which failed. Looking into it.

        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.camel.converter.jaxb.FilteringXmlStreamWriterTest
        Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.333 sec
        Running org.apache.camel.converter.jaxb.JaxbConverterTest
        Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.692 sec
        Running org.apache.camel.converter.jaxb.JaxbDataFormatTest
        Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.497 sec <<< FAILURE!
        Running org.apache.camel.converter.jaxb.MessageTypeTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.624 sec
        Running org.apache.camel.converter.jaxb.NonXmlCharFiltererTest
        Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.354 sec
        Running org.apache.camel.converter.jaxb.NonXmlFilterReaderTest
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.323 sec
        Running org.apache.camel.example.DataFormatConcurrentTest
        Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.689 sec
        Running org.apache.camel.example.DataFormatDataSetTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.838 sec
        Running org.apache.camel.example.DataFormatTest
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.997 sec
        Running org.apache.camel.example.JAXBConvertTest
        Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.632 sec
        Running org.apache.camel.example.JAXBConverterMessageTypeTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.619 sec
        Running org.apache.camel.example.JaxbConcurrentDataFormatTest
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.926 sec
        Running org.apache.camel.example.JaxbErrorLogTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.83 sec
        Running org.apache.camel.example.RouteWithErrorHandlerTest
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.172 sec
        Running org.apache.camel.example.SpringDataFormatPartialTest
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.457 sec
        Running org.apache.camel.example.SpringDataFormatTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.804 sec
        Running org.apache.camel.example.SpringDataFormatWithEncodingTest
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.461 sec
        Running org.apache.camel.example.SpringDataFormatWithMarshalAndUnmarshalUsingRefTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.856 sec
        Running org.apache.camel.example.SpringDataFormatWithMarshalAndUnmarshalUsingSpringBeanTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.888 sec
        Running org.apache.camel.example.UnmarshalTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.84 sec
        Running org.apache.camel.jaxb.CamelJaxbFallbackConverterTest
        Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.801 sec
        Running org.apache.camel.jaxb.CamelJaxbSpringTest
        Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.042 sec
        Running org.apache.camel.jaxb.CamelJaxbTest
        Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.344 sec
        Running org.apache.camel.jaxb.DirectBeanToBeanPersonTypeTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.623 sec
        Running org.apache.camel.jaxb.DumpToXmlTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.621 sec
        Running org.apache.camel.jaxb.FallbackTypeConverterShouldNotThrowExceptionTest
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.099 sec
        Running org.apache.camel.jaxb.JaxbDataFormatIssueTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.877 sec
        Running org.apache.camel.jaxb.JaxbDataFormatIssueUsingSpringJavaConfigTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.477 sec
        Running org.apache.camel.jaxb.TimerBeanToBeanNotificationTypeTest
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.124 sec
        
        Results :
        
        Tests in error: 
        
        Tests run: 79, Failures: 0, Errors: 2, Skipped: 0
        
        
        Show
        Frank Kootte added a comment - I did an update on TRUNK and tried to run the unit tests on camel-jaxb which failed. Looking into it. ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.camel.converter.jaxb.FilteringXmlStreamWriterTest Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.333 sec Running org.apache.camel.converter.jaxb.JaxbConverterTest Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.692 sec Running org.apache.camel.converter.jaxb.JaxbDataFormatTest Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.497 sec <<< FAILURE! Running org.apache.camel.converter.jaxb.MessageTypeTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.624 sec Running org.apache.camel.converter.jaxb.NonXmlCharFiltererTest Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.354 sec Running org.apache.camel.converter.jaxb.NonXmlFilterReaderTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.323 sec Running org.apache.camel.example.DataFormatConcurrentTest Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.689 sec Running org.apache.camel.example.DataFormatDataSetTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.838 sec Running org.apache.camel.example.DataFormatTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.997 sec Running org.apache.camel.example.JAXBConvertTest Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.632 sec Running org.apache.camel.example.JAXBConverterMessageTypeTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.619 sec Running org.apache.camel.example.JaxbConcurrentDataFormatTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.926 sec Running org.apache.camel.example.JaxbErrorLogTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.83 sec Running org.apache.camel.example.RouteWithErrorHandlerTest Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.172 sec Running org.apache.camel.example.SpringDataFormatPartialTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.457 sec Running org.apache.camel.example.SpringDataFormatTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.804 sec Running org.apache.camel.example.SpringDataFormatWithEncodingTest Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.461 sec Running org.apache.camel.example.SpringDataFormatWithMarshalAndUnmarshalUsingRefTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.856 sec Running org.apache.camel.example.SpringDataFormatWithMarshalAndUnmarshalUsingSpringBeanTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.888 sec Running org.apache.camel.example.UnmarshalTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.84 sec Running org.apache.camel.jaxb.CamelJaxbFallbackConverterTest Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.801 sec Running org.apache.camel.jaxb.CamelJaxbSpringTest Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.042 sec Running org.apache.camel.jaxb.CamelJaxbTest Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.344 sec Running org.apache.camel.jaxb.DirectBeanToBeanPersonTypeTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.623 sec Running org.apache.camel.jaxb.DumpToXmlTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.621 sec Running org.apache.camel.jaxb.FallbackTypeConverterShouldNotThrowExceptionTest Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.099 sec Running org.apache.camel.jaxb.JaxbDataFormatIssueTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.877 sec Running org.apache.camel.jaxb.JaxbDataFormatIssueUsingSpringJavaConfigTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.477 sec Running org.apache.camel.jaxb.TimerBeanToBeanNotificationTypeTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.124 sec Results : Tests in error: Tests run: 79, Failures: 0, Errors: 2, Skipped: 0
        Hide
        Frank Kootte added a comment - - edited

        The problem seems to be in the region of the ReentrantLock not being instantiated and the call to doCallRealMethod invoking JaxbDataFormat.unmarshal which depends on the state of the JaxbDataFormat instance.

        java.lang.NullPointerException
        	at org.apache.camel.converter.jaxb.JaxbDataFormat.unmarshal(JaxbDataFormat.java:138)
        	at org.apache.camel.converter.jaxb.JaxbDataFormatTest.testUnmarshalFilteringDisabled(JaxbDataFormatTest.java:152)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
        	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
        	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
        	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
        	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
        	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
        	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
        	at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37)
        	at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62)
        	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
        	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:71)
        	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
        	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
        
        
        Show
        Frank Kootte added a comment - - edited The problem seems to be in the region of the ReentrantLock not being instantiated and the call to doCallRealMethod invoking JaxbDataFormat.unmarshal which depends on the state of the JaxbDataFormat instance. java.lang.NullPointerException at org.apache.camel.converter.jaxb.JaxbDataFormat.unmarshal(JaxbDataFormat.java:138) at org.apache.camel.converter.jaxb.JaxbDataFormatTest.testUnmarshalFilteringDisabled(JaxbDataFormatTest.java:152) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37) at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62) at org.junit.runner.JUnitCore.run(JUnitCore.java:157) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:71) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
        Hide
        Christian Müller added a comment -

        I hat a short mail conversation with Dan:

        You cannot share Unmarshallers. You can share JAXBContext's, but not
        unmarshallers. You do need to create a new unmarshaller (or marshaller) per
        thread (or pool them). With CXF, we create a new unmarshaller per
        request. If you use the XMLStreamReader based unmarshal methods, it
        doesn't need to call into the slow Parser factory code and such which is
        where the main performance bottleneck is. However, it COULD just move
        the bottleneck elsewhere if you aren't careful. CXF pools the
        XMLInputFactory objects (although, if using Woodstox, you don't need to as
        the XMLInputFactory in Woodstox it threadsafe). That basically allows
        avoiding the the entire parser factory things entirely.

        My answer:
        Thanks for the explanation, now it's clear for me.
        We already cache the JAXBContext and share it. Instead of pooling the Unmarshallers, we cache one singe instance (per JaxbDataFormat instance and one for the FallbackTypeConverter) and use a ReentraceLock to make it's only used once at a time. On my machine this was more faster than the pooling solution (using commons-pool).
        We also using Woodstox and cache the XMLInputFactory (unfortunately not in every place). I will spend some time to improve this too.
        I will investigate on the XMLStreamReader unmarshal methods whether they are faster than the current solution.

        Show
        Christian Müller added a comment - I hat a short mail conversation with Dan: You cannot share Unmarshallers. You can share JAXBContext's, but not unmarshallers. You do need to create a new unmarshaller (or marshaller) per thread (or pool them). With CXF, we create a new unmarshaller per request. If you use the XMLStreamReader based unmarshal methods, it doesn't need to call into the slow Parser factory code and such which is where the main performance bottleneck is. However, it COULD just move the bottleneck elsewhere if you aren't careful. CXF pools the XMLInputFactory objects (although, if using Woodstox, you don't need to as the XMLInputFactory in Woodstox it threadsafe). That basically allows avoiding the the entire parser factory things entirely. My answer: Thanks for the explanation, now it's clear for me. We already cache the JAXBContext and share it. Instead of pooling the Unmarshallers, we cache one singe instance (per JaxbDataFormat instance and one for the FallbackTypeConverter) and use a ReentraceLock to make it's only used once at a time. On my machine this was more faster than the pooling solution (using commons-pool). We also using Woodstox and cache the XMLInputFactory (unfortunately not in every place). I will spend some time to improve this too. I will investigate on the XMLStreamReader unmarshal methods whether they are faster than the current solution.
        Hide
        Frank Kootte added a comment -

        Decided to rebuild our app based on 2.10-SNAPSHOT to run performance tests on the patches landed in TRUNK.

        Show
        Frank Kootte added a comment - Decided to rebuild our app based on 2.10-SNAPSHOT to run performance tests on the patches landed in TRUNK.
        Hide
        Christian Müller added a comment -

        Committed r1309146 to fix the failed unit test

        Show
        Christian Müller added a comment - Committed r1309146 to fix the failed unit test
        Hide
        Christian Müller added a comment -

        Committed r1309261
        Updated the FallbackTypeConverter to use XMLStreamReader as first choice type for unmarshalling to avoid calling into the slow Parser factory (as Dan suggested).
        Also added a new type converter from String -> XMLStreamReader
        reduce the time for the concurrent stress test from 3328ms to 2810ms

        Show
        Christian Müller added a comment - Committed r1309261 Updated the FallbackTypeConverter to use XMLStreamReader as first choice type for unmarshalling to avoid calling into the slow Parser factory (as Dan suggested). Also added a new type converter from String -> XMLStreamReader reduce the time for the concurrent stress test from 3328ms to 2810ms
        Hide
        Christian Müller added a comment -

        Committed r1309262
        We now cache the XMLOutputFactory in FallbackTypeConverter because the one we use (Woodstox) is thread safe. We do the same already in our StaxConverter.

        Show
        Christian Müller added a comment - Committed r1309262 We now cache the XMLOutputFactory in FallbackTypeConverter because the one we use (Woodstox) is thread safe. We do the same already in our StaxConverter.
        Hide
        Frank Kootte added a comment -

        Absolutely great work. Due to some environmental issues we have not been able to deploy our new build yet. I will prepare a new build based on the latest patches and keep you informed.

        Show
        Frank Kootte added a comment - Absolutely great work. Due to some environmental issues we have not been able to deploy our new build yet. I will prepare a new build based on the latest patches and keep you informed.
        Hide
        Christian Müller added a comment -

        Committed r1309588
        Now we use the XMLStreamReader as first choice type for unmarshalling in JaxbDataFormat if filtering is not needed (no big performance improvement).
        From my point of view, we are done with a solution which doesn't pool the Unmarshallers.
        Looking forward for your performance test results before I will mark this issue as resolved.

        Show
        Christian Müller added a comment - Committed r1309588 Now we use the XMLStreamReader as first choice type for unmarshalling in JaxbDataFormat if filtering is not needed (no big performance improvement). From my point of view, we are done with a solution which doesn't pool the Unmarshallers. Looking forward for your performance test results before I will mark this issue as resolved.
        Hide
        Daniel Kulp added a comment -

        I've updated the code on trunk to ALWAYS go through the XMLStreamReader pathways which allows complete removal of the locks and unmarshaller pools. Would definitely appreciate some testing with it, particularly with "real world" payloads.

        Show
        Daniel Kulp added a comment - I've updated the code on trunk to ALWAYS go through the XMLStreamReader pathways which allows complete removal of the locks and unmarshaller pools. Would definitely appreciate some testing with it, particularly with "real world" payloads.
        Hide
        Christian Müller added a comment - - edited

        I did a few test an recorded the fastest one:

        Length: 2046:
        =============
        Before:
        testUnmarshallConcurrent() took 14122ms
        testUnmarshallFallbackConcurrent() took 8479ms

        After:
        testUnmarshallConcurrent() took 5196ms
        testUnmarshallFallbackConcurrent() took 2761ms

        Length: 104
        ===========
        Before:
        testUnmarshallConcurrent() took 7281ms
        testUnmarshallFallbackConcurrent() took 4815ms

        After:
        testUnmarshallConcurrent() took 2767ms
        testUnmarshallFallbackConcurrent() took 2458ms

        Best,
        Christian

        Show
        Christian Müller added a comment - - edited I did a few test an recorded the fastest one: Length: 2046: ============= Before: testUnmarshallConcurrent() took 14122ms testUnmarshallFallbackConcurrent() took 8479ms After: testUnmarshallConcurrent() took 5196ms testUnmarshallFallbackConcurrent() took 2761ms Length: 104 =========== Before: testUnmarshallConcurrent() took 7281ms testUnmarshallFallbackConcurrent() took 4815ms After: testUnmarshallConcurrent() took 2767ms testUnmarshallFallbackConcurrent() took 2458ms Best, Christian
        Hide
        Christian Müller added a comment -

        By using a ByteArrayInputStream as payload, and a "warmed up" system (I sent 100 messages before the measurement to warm up the system) I got the following results with the payload of 2046 bytes:

        testUnmarshallConcurrent() took 2202ms (5196ms by using a string as payload and a cold system)
        testUnmarshallFallbackConcurrent() took 1224ms (2761ms by using a string as payload and a cold system)

        testMarshallConcurrent() took 875ms
        testMarshallFallbackConcurrent() took 999ms

        Best,
        Christian

        Show
        Christian Müller added a comment - By using a ByteArrayInputStream as payload, and a "warmed up" system (I sent 100 messages before the measurement to warm up the system) I got the following results with the payload of 2046 bytes: testUnmarshallConcurrent() took 2202ms (5196ms by using a string as payload and a cold system) testUnmarshallFallbackConcurrent() took 1224ms (2761ms by using a string as payload and a cold system) testMarshallConcurrent() took 875ms testMarshallFallbackConcurrent() took 999ms Best, Christian
        Hide
        Frank Kootte added a comment -

        @Christian - my apologies for the radio silence.

        Unfortunately we've been dealing with some customer facing 3rd party gps-chip issues which received priority over everything which did not allow me to pursue the performance testing endeavors. On top of that I was required to merge a feature branch for which we first need to establish a baseline to compare the patch results against. I am back on the topic again and hope to provide you with valuable metrics soon.

        Show
        Frank Kootte added a comment - @Christian - my apologies for the radio silence. Unfortunately we've been dealing with some customer facing 3rd party gps-chip issues which received priority over everything which did not allow me to pursue the performance testing endeavors. On top of that I was required to merge a feature branch for which we first need to establish a baseline to compare the patch results against. I am back on the topic again and hope to provide you with valuable metrics soon.
        Hide
        Christian Müller added a comment -

        Don't worry about it...
        Dan changed this part in a way that we don't need to synchronize it. This solution should scale much better than my first implementation.
        I will merge this change into the camel-2.9.x and camel-2.8.x branch. If your result is a pooling implementation is faster than the current one, we can reopen this issue (if it's already closed) and work further on it. I'm still really interested in your results...

        Best,
        Christian

        Show
        Christian Müller added a comment - Don't worry about it... Dan changed this part in a way that we don't need to synchronize it. This solution should scale much better than my first implementation. I will merge this change into the camel-2.9.x and camel-2.8.x branch. If your result is a pooling implementation is faster than the current one, we can reopen this issue (if it's already closed) and work further on it. I'm still really interested in your results... Best, Christian
        Hide
        Christian Müller added a comment -

        Because Hadrian will start releasing Camel 2.8.5 and 2.9.2 tomorrow, I have merged the changes into these feature branches.
        If you come back with a smarter/faster solution, please feel free to open a new JIRA.

        Best,
        Christian

        Show
        Christian Müller added a comment - Because Hadrian will start releasing Camel 2.8.5 and 2.9.2 tomorrow, I have merged the changes into these feature branches. If you come back with a smarter/faster solution, please feel free to open a new JIRA. Best, Christian
        Hide
        Frank Kootte added a comment -

        Great! Then we can depends on 2.9.2 instead of 2.10-SNAPSHOT which was giving me the creeps
        We will subject 2.9.2 to the load test today - task planned - and I will ping back. Will look into the patch just to satisfy my curiosity

        Show
        Frank Kootte added a comment - Great! Then we can depends on 2.9.2 instead of 2.10-SNAPSHOT which was giving me the creeps We will subject 2.9.2 to the load test today - task planned - and I will ping back. Will look into the patch just to satisfy my curiosity

          People

          • Assignee:
            Christian Müller
            Reporter:
            Claus Ibsen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development