ServiceMix Components
  1. ServiceMix Components
  2. SMXCOMP-19

Either servicemix-bean or servicemix-jms do not function properly.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Blocker Blocker
    • Resolution: Unresolved
    • Affects Version/s: 3.2.3, servicemix-bean-2008.01, servicemix-jms-2008.01
    • Fix Version/s: 2013.02
    • Labels:
      None
    • Environment:

      Windows XP, 4GB mem, jdk 1.6

      Description

      At some point during the 3.3 SNAPSHOT releases, our application stopped working. I did not have a chance to get a concrete case together to illustrate the problem until now. The test case is as follows (client sends 2000 JMS messages):

      Client JMS App -> JMS consumer -> servicemix-bean -> JMS provider

      The attached project works perfectly fine on a 3.3-SNAPSHOT release I have which appears to be from August 22nd. I've never been able to get this sequence to work since, it appears that the servicemix-bean SU will choke after a couple hundred messages and freeze up.

      When I run the described test on my August 22nd servicemix I will end up with 2000 messages having been queued and dequeued from the test.source topic and queued on the test.destination topic.

      When I run the described test on the 3.3 release, I will end up with 2000 message enqueued and dispatched on the test.source topic, but only 576 dequeued. It mentions the other 1474 are in flight.

      If I look at the threads, I see a bunch that are stuck waiting for an exchange (I'm guessing the servicemix-jms component never sends a reply?):

      Name: pool-flow.seda.servicemix-bean-thread-7
      State: WAITING on org.apache.servicemix.jbi.messaging.InOnlyImpl@158ef1f
      Total blocked: 0 Total waited: 1

      Stack trace:
      java.lang.Object.wait(Native Method)
      org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.waitForExchange(DeliveryChannelImpl.java:709)
      org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.sendSync(DeliveryChannelImpl.java:472)
      org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.sendSync(DeliveryChannelImpl.java:442)
      org.apache.servicemix.client.DefaultServiceMixClient.sendSync(DefaultServiceMixClient.java:156)
      test.BeanImpl.onMessageExchange(BeanImpl.java:96)
      org.apache.servicemix.bean.BeanEndpoint.onProviderExchange(BeanEndpoint.java:226)

      • locked org.apache.servicemix.bean.support.Request@1b2a9cf
        org.apache.servicemix.bean.BeanEndpoint.process(BeanEndpoint.java:212)
        org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:600)
        org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:554)
        org.apache.servicemix.common.AsyncBaseLifeCycle.onMessageExchange(AsyncBaseLifeCycle.java:510)
        org.apache.servicemix.common.SyncLifeCycleWrapper.onMessageExchange(SyncLifeCycleWrapper.java:60)
        org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:620)
        org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
        org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
        org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        java.lang.Thread.run(Thread.java:619)

      This seems to be a bug to me since it worked on that one 3.3-SNAPSHOT, but not on any since. The 3.3-SNAPSHOT I have is also before the renaming of components to 2008.01.

      I'll try a few other tests to see if I can narrow down any other cause since I would really like to get upgraded to the full release.

      Attached is a small project that will allow you to reproduce this issue, it's just a modification of the bridge sample with an extra SU in it. Here is all you need to do:

      1. Build the attached project.
      2. Deploy the built bridge-test-sa-3.3.zip
      3. execute "java -jar test-jms-client-1.0.one-jar.jar" from the bridge-test/test-jms-client/target directory.

      1. bridge-test.zip
        13 kB
        Ryan Moquin
      2. using-new-jms-provider.JPG
        131 kB
        Ryan Moquin
      3. using-old-jms-provider.JPG
        124 kB
        Ryan Moquin

        Activity

        Hide
        Ryan Moquin added a comment -

        I think this issue comes from a change in the servicemix-jms that doesn't make it as forgiving about not handling exchanges properly. I haven't had a chance to go back to trying the new jms consumer endpoint, but my assumption is that I'm not quite handling it properly. I think that is more than likely the only problem. I've noticed if you don't mark exchanges done, you will clog up Servicemix and will eventually not be able to process anything. The server will just sit there frozen.

        Show
        Ryan Moquin added a comment - I think this issue comes from a change in the servicemix-jms that doesn't make it as forgiving about not handling exchanges properly. I haven't had a chance to go back to trying the new jms consumer endpoint, but my assumption is that I'm not quite handling it properly. I think that is more than likely the only problem. I've noticed if you don't mark exchanges done, you will clog up Servicemix and will eventually not be able to process anything. The server will just sit there frozen.
        Hide
        Brian Powell added a comment -

        Hello,

        I was wondering if you were able to find a resolution to this issue. I am also experiencing this.

        Thanks,

        Brian

        Quoted from:
        http://www.nabble.com/-jira--Created%3A-%28SM-1666%29-Either-servicemix-bean-or-servicemix-jms-do-not-function-properly.-tp20240265p20240265.html

        Show
        Brian Powell added a comment - Hello, I was wondering if you were able to find a resolution to this issue. I am also experiencing this. Thanks, Brian Quoted from: http://www.nabble.com/-jira--Created%3A-%28SM-1666%29-Either-servicemix-bean-or-servicemix-jms-do-not-function-properly.-tp20240265p20240265.html
        Hide
        Guillaume Nodet added a comment -

        I think the problems really comes from your code deployed onto servicemix-bean.
        When an exchange comes to an endpoint, the only case when you can ignore those are when those have a DONE or ERROR status.
        If the status ACTIVE, the exchange has to be sent back with an out message, a fault or a DONE / ERROR status depending on the MEP.
        In your code, when the exchange is received and contains an In message, you create a new exchange, populate it and send it, but the original exchange is not sent back.

        Take a look at the following code for example:
        https://svn.apache.org/repos/asf/servicemix/components/engines/servicemix-bean/trunk/src/test/java/org/apache/servicemix/bean/beans/ConsumerListener.java

        It sets the original exchange has a property on the new exchange, so that it can be retrieved and set with a DONE status at a later time. You could also keep the original exchange in a Map or a Store as done on various EIP patterns.

        Show
        Guillaume Nodet added a comment - I think the problems really comes from your code deployed onto servicemix-bean. When an exchange comes to an endpoint, the only case when you can ignore those are when those have a DONE or ERROR status. If the status ACTIVE, the exchange has to be sent back with an out message, a fault or a DONE / ERROR status depending on the MEP. In your code, when the exchange is received and contains an In message, you create a new exchange, populate it and send it, but the original exchange is not sent back. Take a look at the following code for example: https://svn.apache.org/repos/asf/servicemix/components/engines/servicemix-bean/trunk/src/test/java/org/apache/servicemix/bean/beans/ConsumerListener.java It sets the original exchange has a property on the new exchange, so that it can be retrieved and set with a DONE status at a later time. You could also keep the original exchange in a Map or a Store as done on various EIP patterns.
        Hide
        Ryan Moquin added a comment -

        I made the changes you mentioned Guillaume (though I'm using a client created with the ComponentContext because the DeliveryChannel gives me a null NormalizedMessage when calling getInMessage()), and I still see the same results with the new provider endpoint (I didn't bother trying the new jms consumer endpoint, since that doesn't seem to ever work) with the release of 3.3. Remember, I see this same setup working with the 3.3 SNAPSHOT. I attached screenshots this time of my jconsole output so you can see how servicemix freezes when using the new provider endpoint. I have to force quit servicemix when using the new provider endpoint because it completely locks all the threads.

        I also switched the example to be asynchrynchronous. As a result, I now see a big difference in the output. When using the old servicemix-jms provider, I see the following line outputted:

        Exchange was marked done, not doing anything else.

        When I switch to the new servicemix-jms provider (with no other changes), I do NOT see that line anymore. Which means it's not responding with a done, and therefore would explain how all the threads get tied up.

        Is there anything else I might be doing wrong? It seems like no matter how I structure my code, I always get the same result when testing both endpoint types.

        Show
        Ryan Moquin added a comment - I made the changes you mentioned Guillaume (though I'm using a client created with the ComponentContext because the DeliveryChannel gives me a null NormalizedMessage when calling getInMessage()), and I still see the same results with the new provider endpoint (I didn't bother trying the new jms consumer endpoint, since that doesn't seem to ever work) with the release of 3.3. Remember, I see this same setup working with the 3.3 SNAPSHOT. I attached screenshots this time of my jconsole output so you can see how servicemix freezes when using the new provider endpoint. I have to force quit servicemix when using the new provider endpoint because it completely locks all the threads. I also switched the example to be asynchrynchronous. As a result, I now see a big difference in the output. When using the old servicemix-jms provider, I see the following line outputted: Exchange was marked done, not doing anything else. When I switch to the new servicemix-jms provider (with no other changes), I do NOT see that line anymore. Which means it's not responding with a done, and therefore would explain how all the threads get tied up. Is there anything else I might be doing wrong? It seems like no matter how I structure my code, I always get the same result when testing both endpoint types.
        Hide
        Ryan Moquin added a comment -

        Code changes recommended by Guillaume to prevent threading issues with the Transformer class and to not use the ServiceMixClient looked up in JNDI, vs. service resources.

        Show
        Ryan Moquin added a comment - Code changes recommended by Guillaume to prevent threading issues with the Transformer class and to not use the ServiceMixClient looked up in JNDI, vs. service resources.
        Hide
        Ryan Moquin added a comment -

        This is a screenshot using the old jms provider endpoint and the code adjustments recommended for my test. Notice that the test.destination topic is listed (vs. using the new provider endpoint).

        Show
        Ryan Moquin added a comment - This is a screenshot using the old jms provider endpoint and the code adjustments recommended for my test. Notice that the test.destination topic is listed (vs. using the new provider endpoint).
        Hide
        Ryan Moquin added a comment -

        This is a screenshot using the new jms provider endpoint and the code adjustments recommended for my test. Notice that the test.destination topic is not listed even though all 2000 messages have been sent from the external client.

        Show
        Ryan Moquin added a comment - This is a screenshot using the new jms provider endpoint and the code adjustments recommended for my test. Notice that the test.destination topic is not listed even though all 2000 messages have been sent from the external client.
        Hide
        Ryan Moquin added a comment -

        Go figure that my working circumstance didn't exhibit any of this behavior in a way that I recognized as being the same thing. I guess I was thinking about how JAXB is threadsafe and that led me to believe that the transformer would also be threadsafe.

        I seemed to have some weird problems with the deliveryChannel when sending out, which was why I had stopped using it. Maybe that was a result of this problem as well. I'll give it a try.

        Show
        Ryan Moquin added a comment - Go figure that my working circumstance didn't exhibit any of this behavior in a way that I recognized as being the same thing. I guess I was thinking about how JAXB is threadsafe and that led me to believe that the transformer would also be threadsafe. I seemed to have some weird problems with the deliveryChannel when sending out, which was why I had stopped using it. Maybe that was a result of this problem as well. I'll give it a try.
        Hide
        Guillaume Nodet added a comment -

        The problem may actually come from your test bean.
        The javax.transform.Transformer object is not thread safe, and you are using a single instance to process all the concurrent exchanges coming to your bean.
        This may very well lead to the kind of problems you report, as the generated xml may be completely messed up.
        Could you try with creating a transformer for each message or using a synchronized block.

        Also, when sending exchanges from an endpoint, the use of a client is not really good as you will bypass some processing and won't be really able to send the exchanges asynchronously. Instead, you should use the injected DeliveryChannel to create and send the exchange.

        Show
        Guillaume Nodet added a comment - The problem may actually come from your test bean. The javax.transform.Transformer object is not thread safe, and you are using a single instance to process all the concurrent exchanges coming to your bean. This may very well lead to the kind of problems you report, as the generated xml may be completely messed up. Could you try with creating a transformer for each message or using a synchronized block. Also, when sending exchanges from an endpoint, the use of a client is not really good as you will bypass some processing and won't be really able to send the exchanges asynchronously. Instead, you should use the injected DeliveryChannel to create and send the exchange.
        Hide
        Ryan Moquin added a comment -

        Correct a small mistake where I was sending empty messages rather than the intended message.

        Show
        Ryan Moquin added a comment - Correct a small mistake where I was sending empty messages rather than the intended message.
        Hide
        Ryan Moquin added a comment -

        Another interesting tidbit. So this example works on that 3.3-SNAPSHOT, but not on the 3.3 release. If I switch the JMS provider endpoint to the old jms provider endpoint and run the test again, the test.destination ends up with 1998 messages, 2 are missing. When I look back at the servicemix output, I see it complain that one of the xml message I'm adding as content is invalid:

        [Fatal Error] :1:70: The element type "testMessage1" must be terminated by the matching end-tag "</testMessage1>".

        But my message doesn't have a testMessage1 as part of it's tag, so obviously the message it's complaining about was modified somehow. Should servicemix-bean components always be synchronized? I never seemed to have this problem before unless I was doing something that could be intensive.

        This is how I'm constructing my message, and in my client output the messages are all correct.
        String message = "<testMessage>" + i + "</testMessage>";

        When I ran the test a second and third time, it looks like all the messages made it through without an error. It appears the new servicemix-jms endpoints will cause the system to lock up because it doesn't return an acknowledgement? With the old endpoints, they appear to spit out the error and drop the message, so at least things don't hang that way. Also, I noticed that I was sending blank messages to the jms provider in the attached sample, which apparently used to work in the old 3.3 (it's not what I'm doing in my actual app), but that causes errors in the newest 3.3 versions as well. I corrected the test code and am reattaching it (it has the old provider endpoint added as well).

        Should the servicemix-jms endpoint fail graceful when a problem occurs? If I exit servicemix when the new end point is hanging, the console will scroll for a minute or two with a ton of error output from the endpoint. I guess I'll discontinue my use of the newest servicemix-jms enpoints until it's known why they are problematic.

        Here is a snippet of the log:

        [Fatal Error] :-1:-1: Premature end of file.
        [Fatal Error] :1:70: The element type "testMessage1" must be terminated by the matching end-tag "</testMessage1>".
        ERROR - JmsComponent - Error processing exchange InOnly[
        id: ID:192.168.1.4-11d4ed3c406-13:0
        status: Active
        role: provider
        service:

        {http://servicemix.apache.org/samples/bridge}

        jmstwo
        endpoint: endpoint
        in: Unable to display: org.xml.sax.SAXParseException: The element type "testMessage1" must be terminated by the match
        g end-tag "</testMessage1>".
        ]
        com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </testMessage>; expected </testMessage1>.
        at [row,col

        {unknown-source}

        ]: [1,80]
        at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:605)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:461)
        at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3256)
        at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3198)

        Show
        Ryan Moquin added a comment - Another interesting tidbit. So this example works on that 3.3-SNAPSHOT, but not on the 3.3 release. If I switch the JMS provider endpoint to the old jms provider endpoint and run the test again, the test.destination ends up with 1998 messages, 2 are missing. When I look back at the servicemix output, I see it complain that one of the xml message I'm adding as content is invalid: [Fatal Error] :1:70: The element type "testMessage1" must be terminated by the matching end-tag "</testMessage1>". But my message doesn't have a testMessage1 as part of it's tag, so obviously the message it's complaining about was modified somehow. Should servicemix-bean components always be synchronized? I never seemed to have this problem before unless I was doing something that could be intensive. This is how I'm constructing my message, and in my client output the messages are all correct. String message = "<testMessage>" + i + "</testMessage>"; When I ran the test a second and third time, it looks like all the messages made it through without an error. It appears the new servicemix-jms endpoints will cause the system to lock up because it doesn't return an acknowledgement? With the old endpoints, they appear to spit out the error and drop the message, so at least things don't hang that way. Also, I noticed that I was sending blank messages to the jms provider in the attached sample, which apparently used to work in the old 3.3 (it's not what I'm doing in my actual app), but that causes errors in the newest 3.3 versions as well. I corrected the test code and am reattaching it (it has the old provider endpoint added as well). Should the servicemix-jms endpoint fail graceful when a problem occurs? If I exit servicemix when the new end point is hanging, the console will scroll for a minute or two with a ton of error output from the endpoint. I guess I'll discontinue my use of the newest servicemix-jms enpoints until it's known why they are problematic. Here is a snippet of the log: [Fatal Error] :-1:-1: Premature end of file. [Fatal Error] :1:70: The element type "testMessage1" must be terminated by the matching end-tag "</testMessage1>". ERROR - JmsComponent - Error processing exchange InOnly[ id: ID:192.168.1.4-11d4ed3c406-13:0 status: Active role: provider service: {http://servicemix.apache.org/samples/bridge} jmstwo endpoint: endpoint in: Unable to display: org.xml.sax.SAXParseException: The element type "testMessage1" must be terminated by the match g end-tag "</testMessage1>". ] com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </testMessage>; expected </testMessage1>. at [row,col {unknown-source} ]: [1,80] at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:605) at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:461) at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3256) at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3198)
        Hide
        Ryan Moquin added a comment -

        One more curious thing, is that in the bridge-jms-one-su SU of the attached maven2 project, the xbean.xml contains the old servicemix-jms consumer endpoint and a commented new servicemix-jms consumer endpoint. If I use the new servicemix-jms consumer endpoint, I will then see this sample project fail even on the 3.3-SNAPSHOT that works with the old servicemix-jms consumer endpoint. With the new endpoint, the test.source topic will enqueue all 2000 messages, but on 1 will be dequeued. As a result only 1 message makes it to the test.destination. So it appears that this locks up with the new servicemix-jms consumer endpoint, whereas it runs perfectly fine with the old servicemix-jms consumer endpoint. It seems like these two conditions are related somehow.

        Show
        Ryan Moquin added a comment - One more curious thing, is that in the bridge-jms-one-su SU of the attached maven2 project, the xbean.xml contains the old servicemix-jms consumer endpoint and a commented new servicemix-jms consumer endpoint. If I use the new servicemix-jms consumer endpoint, I will then see this sample project fail even on the 3.3-SNAPSHOT that works with the old servicemix-jms consumer endpoint. With the new endpoint, the test.source topic will enqueue all 2000 messages, but on 1 will be dequeued. As a result only 1 message makes it to the test.destination. So it appears that this locks up with the new servicemix-jms consumer endpoint, whereas it runs perfectly fine with the old servicemix-jms consumer endpoint. It seems like these two conditions are related somehow.

          People

          • Assignee:
            Unassigned
            Reporter:
            Ryan Moquin
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development