Camel
  1. Camel
  2. CAMEL-6229

InOut ActiveMQ exception Cannot publish to a deleted Destination

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      When exposing a cxf-rs webservice and attempting to talk to another route, using InOut MEP + ActiveMQ the following exception occurs

      22:18:14,813 | INFO  | tp1882786420-364 | route1                           | 147 - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Received a request :: 
      22:18:14,838 | WARN  | nager[temporary] | faultJmsMessageListenerContainer | 153 - org.springframework.jms - 3.0.7.RELEASE | Setup of JMS message listener invoker failed for destination 'temporary' - trying to recover. Cause: The Consumer is closed
      22:18:14,849 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:3" on closing pooled connection: The connection is already closed
      22:18:14,850 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:4" on closing pooled connection: The connection is already closed
      22:18:14,850 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:5" on closing pooled connection: The connection is already closed
      22:18:14,850 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:6" on closing pooled connection: The connection is already closed
      22:18:14,850 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:7" on closing pooled connection: The connection is already closed
      22:18:14,850 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:8" on closing pooled connection: The connection is already closed
      22:18:14,851 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:2" on closing pooled connection: The connection is already closed
      22:18:14,851 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:1" on closing pooled connection: The connection is already closed
      22:18:14,851 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:9" on closing pooled connection: The connection is already closed
      22:18:14,851 | INFO  | nager[temporary] | PooledConnection                 | 122 - org.apache.activemq.activemq-pool - 5.7.0.fuse-71-047 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-49913-1364764601861-3:3:10" on closing pooled connection: The connection is already closed
      22:18:14,857 | WARN  | responseHandler] | faultJmsMessageListenerContainer | 153 - org.springframework.jms - 3.0.7.RELEASE | Setup of JMS message listener invoker failed for destination 'responseHandler' - trying to recover. Cause: The Consumer is closed
      22:18:14,919 | INFO  | responseHandler] | faultJmsMessageListenerContainer | 153 - org.springframework.jms - 3.0.7.RELEASE | Successfully refreshed JMS Connection
      22:18:14,927 | INFO  | nager[temporary] | faultJmsMessageListenerContainer | 153 - org.springframework.jms - 3.0.7.RELEASE | Successfully refreshed JMS Connection
      22:18:14,939 | INFO  | responseHandler] | route2                           | 147 - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Response handler successfully received a request!
      22:18:14,943 | WARN  | responseHandler] | EndpointMessageListener          | 147 - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Execution of JMS message listener failed. Caused by: [org.apache.camel.RuntimeCamelException - org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:alan-dell-49913-1364764601861-3:3:10; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:alan-dell-49913-1364764601861-3:3:10]
      org.apache.camel.RuntimeCamelException: org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:alan-dell-49913-1364764601861-3:3:10; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:alan-dell-49913-1364764601861-3:3:10
      	at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1271)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:125)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)[153:org.springframework.jms:3.0.7.RELEASE]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)[:1.6.0_27]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)[:1.6.0_27]
      	at java.lang.Thread.run(Thread.java:662)[:1.6.0_27]
      Caused by: org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:alan-dell-49913-1364764601861-3:3:10; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:alan-dell-49913-1364764601861-3:3:10
      	at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:285)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:200)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.EndpointMessageListener.sendReply(EndpointMessageListener.java:358)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback.done(EndpointMessageListener.java:211)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:108)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	... 11 more
      Caused by: javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:alan-dell-49913-1364764601861-3:3:10
      	at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1731)[125:org.apache.activemq.activemq-core:5.7.0.fuse-71-047]
      	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)[125:org.apache.activemq.activemq-core:5.7.0.fuse-71-047]
      	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)[125:org.apache.activemq.activemq-core:5.7.0.fuse-71-047]
      	at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:79)[122:org.apache.activemq.activemq-pool:5.7.0.fuse-71-047]
      	at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:62)[122:org.apache.activemq.activemq-pool:5.7.0.fuse-71-047]
      	at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:592)[153:org.springframework.jms:3.0.7.RELEASE]
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSend(JmsConfiguration.java:278)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:217)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:159)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$4.doInJms(JmsConfiguration.java:202)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)[153:org.springframework.jms:3.0.7.RELEASE]
      	... 15 more
      22:18:35,016 | WARN  | responseHandler] | TemporaryQueueReplyManager       | 154 - org.apache.camel.camel-jms - 2.10.0.fuse-71-047 | Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-alan-dell-49919-1364764620418-0-215]. Setting ExchangeTimedOutException on (MessageId: ID-alan-dell-49919-1364764620418-0-214 on ExchangeId: ID-alan-dell-49919-1364764620418-0-213) and continue routing.
      22:18:35,019 | ERROR | responseHandler] | DefaultErrorHandler              | 147 - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Failed delivery for (MessageId: ID-alan-dell-49919-1364764620418-0-214 on ExchangeId: ID-alan-dell-49919-1364764620418-0-213). Exhausted after delivery attempt: 1 caught: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 20000 millis due reply message with correlationID: Camel-ID-alan-dell-49919-1364764620418-0-215 not received. Exchange[Message: [foo.bar.rest.Request@2f417b29]]
      org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 20000 millis due reply message with correlationID: Camel-ID-alan-dell-49919-1364764620418-0-215 not received. Exchange[Message: [foo.bar.rest.Request@2f417b29]]
      	at org.apache.camel.component.jms.reply.ReplyManagerSupport.processReply(ReplyManagerSupport.java:133)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler.onTimeout(TemporaryQueueReplyHandler.java:61)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.reply.CorrelationTimeoutMap.onEviction(CorrelationTimeoutMap.java:53)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.component.jms.reply.CorrelationTimeoutMap.onEviction(CorrelationTimeoutMap.java:30)[154:org.apache.camel.camel-jms:2.10.0.fuse-71-047]
      	at org.apache.camel.support.DefaultTimeoutMap.purge(DefaultTimeoutMap.java:209)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
      	at org.apache.camel.support.DefaultTimeoutMap.run(DefaultTimeoutMap.java:159)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)[:1.6.0_27]
      	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)[:1.6.0_27]
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)[:1.6.0_27]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)[:1.6.0_27]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)[:1.6.0_27]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)[:1.6.0_27]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)[:1.6.0_27]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)[:1.6.0_27]
      	at java.lang.Thread.run(Thread.java:662)[:1.6.0_27]
      22:18:35,030 | WARN  | tp1882786420-362 | WebApplicationExceptionMapper    | 183 - org.apache.cxf.cxf-rt-frontend-jaxrs - 2.6.0.fuse-71-047 | javax.ws.rs.WebApplicationException: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 20000 millis due reply message with correlationID: Camel-ID-alan-dell-49919-1364764620418-0-215 not received. Exchange[Message: [foo.bar.rest.Request@2f417b29]]
      	at org.apache.camel.component.cxf.jaxrs.CxfRsInvoker.returnResponse(CxfRsInvoker.java:149)
      	at org.apache.camel.component.cxf.jaxrs.CxfRsInvoker.asyncInvoke(CxfRsInvoker.java:104)
      	at org.apache.camel.component.cxf.jaxrs.CxfRsInvoker.performInvocation(CxfRsInvoker.java:57)
      	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
      	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167)
      	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94)
      	at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
      	at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94)
      	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
      	at org.apache.cxf.phase.PhaseInterceptorChain.resume(PhaseInterceptorChain.java:231)
      	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:78)
      	at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:348)
      	at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:312)
      	at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:72)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1040)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:976)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:193)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handleAsync(Server.java:403)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:488)
      	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:79)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 20000 millis due reply message with correlationID: Camel-ID-alan-dell-49919-1364764620418-0-215 not received. Exchange[Message: [foo.bar.rest.Request@2f417b29]]
      	at org.apache.camel.component.jms.reply.ReplyManagerSupport.processReply(ReplyManagerSupport.java:133)
      	at org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler.onTimeout(TemporaryQueueReplyHandler.java:61)
      	at org.apache.camel.component.jms.reply.CorrelationTimeoutMap.onEviction(CorrelationTimeoutMap.java:53)
      	at org.apache.camel.component.jms.reply.CorrelationTimeoutMap.onEviction(CorrelationTimeoutMap.java:30)
      	at org.apache.camel.support.DefaultTimeoutMap.purge(DefaultTimeoutMap.java:209)
      	at org.apache.camel.support.DefaultTimeoutMap.run(DefaultTimeoutMap.java:159)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	... 1 more
      

      Using :
      Camel 2.10
      ActiveMQ Camel 5.7
      cxf 2.6
      Windows 7

      1. camel-6229.rar
        6 kB
        Alan Foster

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Closed Closed
          14h 11m 1 Alan Foster 01/Apr/13 12:45
          Gavin made changes -
          Link This issue relates to CAMEL-5865 [ CAMEL-5865 ]
          Gavin made changes -
          Link This issue relates to CAMEL-5865 [ CAMEL-5865 ]
          Alan Foster made changes -
          Status Open [ 1 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Alan Foster made changes -
          Link This issue relates CAMEL-5865 [ CAMEL-5865 ]
          Hide
          Alan Foster added a comment -

          Raúl Kripalani Ah, those conditions do seem to line up with what I am using currently;
          I was installing this code using osgi:install -s mvn:org/camel/1.0-SNAPSHOT and not the hot deploy directory

          I'll close this issue though as you have already fixed this bug in CAMEL-5865, thanks!

          Show
          Alan Foster added a comment - Raúl Kripalani Ah, those conditions do seem to line up with what I am using currently; I was installing this code using osgi:install -s mvn:org/camel/1.0-SNAPSHOT and not the hot deploy directory I'll close this issue though as you have already fixed this bug in CAMEL-5865 , thanks!
          Hide
          Raúl Kripalani added a comment -

          Alan Foster - I've reproduced this situation in the past when using the VM Transport and SMX 4.5.0 or above. In my case, the culprit was the activemq-broker.xml hotdeploy feature, which was redeploying the broker soon after the Camel routes started, therefore leaving them in an inconsistent state.

          Try removing this file: SMX_HOME/etc/org.apache.felix.fileinstall-activemq.cfg, deleting the data/ directory and starting afresh.

          Also, installing bundles by copying them to the deploy/ directory could be another cause, as Karaf refreshes all linked bundles during the process (possibly the broker too). How do you perform the deployment?

          Show
          Raúl Kripalani added a comment - Alan Foster - I've reproduced this situation in the past when using the VM Transport and SMX 4.5.0 or above. In my case, the culprit was the activemq-broker.xml hotdeploy feature, which was redeploying the broker soon after the Camel routes started, therefore leaving them in an inconsistent state. Try removing this file: SMX_HOME/etc/org.apache.felix.fileinstall-activemq.cfg, deleting the data/ directory and starting afresh. Also, installing bundles by copying them to the deploy/ directory could be another cause, as Karaf refreshes all linked bundles during the process (possibly the broker too). How do you perform the deployment?
          Hide
          Alan Foster added a comment - - edited

          Ah, I was just about to mark this as resolved, and then I noticed some very odd things in the log files :

          0:28:59,458 | INFO  | tp1937409810-134 | route1                           | 96 - org.apache.camel.camel-core - 2.10.4 | Received a request :: 
          00:28:59,461 | WARN  | responseHandler] | TemporaryQueueReplyManager       | 125 - org.apache.camel.camel-jms - 2.10.4 | Exception inside the DMLC for Temporary ReplyTo Queue for destination responseHandler, refreshing ReplyTo destination
          javax.jms.IllegalStateException: The Consumer is closed
          	at org.apache.activemq.ActiveMQMessageConsumer.checkClosed(ActiveMQMessageConsumer.java:817)[62:org.apache.activemq.activemq-core:5.7.0]
          	at org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:581)[62:org.apache.activemq.activemq-core:5.7.0]
          	at org.apache.activemq.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)[59:org.apache.activemq.activemq-pool:5.7.0]
          	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)[123:org.springframework.jms:3.0.7.RELEASE]
          	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)[123:org.springframework.jms:3.0.7.RELEASE]
          	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)[123:org.springframework.jms:3.0.7.RELEASE]
          	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)[123:org.springframework.jms:3.0.7.RELEASE]
          	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)[123:org.springframework.jms:3.0.7.RELEASE]
          	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)[123:org.springframework.jms:3.0.7.RELEASE]
          	at java.lang.Thread.run(Thread.java:662)[:1.6.0_27]
          00:28:59,462 | WARN  | responseHandler] | faultJmsMessageListenerContainer | 123 - org.springframework.jms - 3.0.7.RELEASE | Setup of JMS message listener invoker failed for destination 'temporary' - trying to recover. Cause: The Consumer is closed
          00:28:59,468 | INFO  | responseHandler] | PooledConnection                 | 59 - org.apache.activemq.activemq-pool - 5.7.0 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-53847-1364772340736-5:17:1" on closing pooled connection: The connection is already closed
          00:28:59,472 | INFO  | responseHandler] | PooledConnection                 | 59 - org.apache.activemq.activemq-pool - 5.7.0 | failed to delete Temporary Queue "temp-queue://ID:alan-dell-53847-1364772340736-5:17:2" on closing pooled connection: The connection is already closed
          00:28:59,472 | WARN  | responseHandler] | faultJmsMessageListenerContainer | 123 - org.springframework.jms - 3.0.7.RELEASE | Setup of JMS message listener invoker failed for destination 'responseHandler' - trying to recover. Cause: The Session is closed
          00:28:59,479 | WARN  | responseHandler] | BrokerRegistry                   | 62 - org.apache.activemq.activemq-core - 5.7.0 | Broker localhost not started so using default instead
          00:28:59,482 | WARN  | responseHandler] | BrokerRegistry                   | 62 - org.apache.activemq.activemq-core - 5.7.0 | Broker localhost not started so using default instead
          00:28:59,486 | INFO  | responseHandler] | faultJmsMessageListenerContainer | 123 - org.springframework.jms - 3.0.7.RELEASE | Successfully refreshed JMS Connection
          

          And these are continuously output the log file too

          00:32:11,049 | WARN  | responseHandler] | TemporaryQueueReplyManager       | 125 - org.apache.camel.camel-jms - 2.10.4 | Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-alan-dell-53850-1364772369222-0-126]. Setting ExchangeTimedOutException on (MessageId: ID-alan-dell-53850-1364772369222-0-125 on ExchangeId: ID-alan-dell-53850-1364772369222-0-124) and continue routing.
          00:32:11,053 | WARN  | responseHandler] | CorrelationTimeoutMap            | 96 - org.apache.camel.camel-core - 2.10.4 | Exception happened during eviction of entry ID org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler@4efdd8b, won't evict and will continue trying: java.lang.IllegalStateException: COMPLETE,resumed
          00:32:12,055 | WARN  | responseHandler] | TemporaryQueueReplyManager       | 125 - org.apache.camel.camel-jms - 2.10.4 | Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-alan-dell-53850-1364772369222-0-126]. Setting ExchangeTimedOutException on (MessageId: ID-alan-dell-53850-1364772369222-0-125 on ExchangeId: ID-alan-dell-53850-1364772369222-0-124) and continue routing.
          00:32:12,056 | WARN  | responseHandler] | CorrelationTimeoutMap            | 96 - org.apache.camel.camel-core - 2.10.4 | Exception happened during eviction of entry ID org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler@4efdd8b, won't evict and will continue trying: java.lang.IllegalStateException: COMPLETE,resumed
          00:32:13,057 | WARN  | responseHandler] | TemporaryQueueReplyManager       | 125 - org.apache.camel.camel-jms - 2.10.4 | Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-alan-dell-53850-1364772369222-0-126]. Setting ExchangeTimedOutException on (MessageId: ID-alan-dell-53850-1364772369222-0-125 on ExchangeId: ID-alan-dell-53850-1364772369222-0-124) and continue routing.
          00:32:13,058 | WARN  | responseHandler] | CorrelationTimeoutMap            | 96 - org.apache.camel.camel-core - 2.10.4 | Exception happened during eviction of entry ID org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler@4efdd8b, won't evict and will continue trying: java.lang.IllegalStateException: COMPLETE,resumed
          

          I'm also seeing a lot of other exceptions in the log file too, related to both cxf/camel/activemq, but it seems to handle new requests just fine!

          I wonder if this is reproducible on your end at all?

          I really wish I could've gotten that JUnit test to run!

          Edit - I can't seem to reproduce this after a fresh start of servicemix unfortunately... But something did go very wrong!

          Show
          Alan Foster added a comment - - edited Ah, I was just about to mark this as resolved, and then I noticed some very odd things in the log files : 0:28:59,458 | INFO | tp1937409810-134 | route1 | 96 - org.apache.camel.camel-core - 2.10.4 | Received a request :: 00:28:59,461 | WARN | responseHandler] | TemporaryQueueReplyManager | 125 - org.apache.camel.camel-jms - 2.10.4 | Exception inside the DMLC for Temporary ReplyTo Queue for destination responseHandler, refreshing ReplyTo destination javax.jms.IllegalStateException: The Consumer is closed at org.apache.activemq.ActiveMQMessageConsumer.checkClosed(ActiveMQMessageConsumer.java:817)[62:org.apache.activemq.activemq-core:5.7.0] at org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:581)[62:org.apache.activemq.activemq-core:5.7.0] at org.apache.activemq.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)[59:org.apache.activemq.activemq-pool:5.7.0] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)[123:org.springframework.jms:3.0.7.RELEASE] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)[123:org.springframework.jms:3.0.7.RELEASE] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)[123:org.springframework.jms:3.0.7.RELEASE] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)[123:org.springframework.jms:3.0.7.RELEASE] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)[123:org.springframework.jms:3.0.7.RELEASE] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)[123:org.springframework.jms:3.0.7.RELEASE] at java.lang. Thread .run( Thread .java:662)[:1.6.0_27] 00:28:59,462 | WARN | responseHandler] | faultJmsMessageListenerContainer | 123 - org.springframework.jms - 3.0.7.RELEASE | Setup of JMS message listener invoker failed for destination 'temporary' - trying to recover. Cause: The Consumer is closed 00:28:59,468 | INFO | responseHandler] | PooledConnection | 59 - org.apache.activemq.activemq-pool - 5.7.0 | failed to delete Temporary Queue "temp-queue: //ID:alan-dell-53847-1364772340736-5:17:1" on closing pooled connection: The connection is already closed 00:28:59,472 | INFO | responseHandler] | PooledConnection | 59 - org.apache.activemq.activemq-pool - 5.7.0 | failed to delete Temporary Queue "temp-queue: //ID:alan-dell-53847-1364772340736-5:17:2" on closing pooled connection: The connection is already closed 00:28:59,472 | WARN | responseHandler] | faultJmsMessageListenerContainer | 123 - org.springframework.jms - 3.0.7.RELEASE | Setup of JMS message listener invoker failed for destination 'responseHandler' - trying to recover. Cause: The Session is closed 00:28:59,479 | WARN | responseHandler] | BrokerRegistry | 62 - org.apache.activemq.activemq-core - 5.7.0 | Broker localhost not started so using default instead 00:28:59,482 | WARN | responseHandler] | BrokerRegistry | 62 - org.apache.activemq.activemq-core - 5.7.0 | Broker localhost not started so using default instead 00:28:59,486 | INFO | responseHandler] | faultJmsMessageListenerContainer | 123 - org.springframework.jms - 3.0.7.RELEASE | Successfully refreshed JMS Connection And these are continuously output the log file too 00:32:11,049 | WARN | responseHandler] | TemporaryQueueReplyManager | 125 - org.apache.camel.camel-jms - 2.10.4 | Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-alan-dell-53850-1364772369222-0-126]. Setting ExchangeTimedOutException on (MessageId: ID-alan-dell-53850-1364772369222-0-125 on ExchangeId: ID-alan-dell-53850-1364772369222-0-124) and continue routing. 00:32:11,053 | WARN | responseHandler] | CorrelationTimeoutMap | 96 - org.apache.camel.camel-core - 2.10.4 | Exception happened during eviction of entry ID org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler@4efdd8b, won't evict and will continue trying: java.lang.IllegalStateException: COMPLETE,resumed 00:32:12,055 | WARN | responseHandler] | TemporaryQueueReplyManager | 125 - org.apache.camel.camel-jms - 2.10.4 | Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-alan-dell-53850-1364772369222-0-126]. Setting ExchangeTimedOutException on (MessageId: ID-alan-dell-53850-1364772369222-0-125 on ExchangeId: ID-alan-dell-53850-1364772369222-0-124) and continue routing. 00:32:12,056 | WARN | responseHandler] | CorrelationTimeoutMap | 96 - org.apache.camel.camel-core - 2.10.4 | Exception happened during eviction of entry ID org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler@4efdd8b, won't evict and will continue trying: java.lang.IllegalStateException: COMPLETE,resumed 00:32:13,057 | WARN | responseHandler] | TemporaryQueueReplyManager | 125 - org.apache.camel.camel-jms - 2.10.4 | Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-alan-dell-53850-1364772369222-0-126]. Setting ExchangeTimedOutException on (MessageId: ID-alan-dell-53850-1364772369222-0-125 on ExchangeId: ID-alan-dell-53850-1364772369222-0-124) and continue routing. 00:32:13,058 | WARN | responseHandler] | CorrelationTimeoutMap | 96 - org.apache.camel.camel-core - 2.10.4 | Exception happened during eviction of entry ID org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler@4efdd8b, won't evict and will continue trying: java.lang.IllegalStateException: COMPLETE,resumed I'm also seeing a lot of other exceptions in the log file too, related to both cxf/camel/activemq, but it seems to handle new requests just fine! I wonder if this is reproducible on your end at all? I really wish I could've gotten that JUnit test to run! Edit - I can't seem to reproduce this after a fresh start of servicemix unfortunately... But something did go very wrong!
          Hide
          Alan Foster added a comment -

          Raúl Kripalani Thanks for the response! I did think that your fixed jira CAMEL-5865 seemed familiar

          I wasn't able to swap out the camel-jms in the fuse container, but I downloaded service mix 4.5.1 which has camel 2.10.4 in it by default, and it seems to be working, hurray

          Show
          Alan Foster added a comment - Raúl Kripalani Thanks for the response! I did think that your fixed jira CAMEL-5865 seemed familiar I wasn't able to swap out the camel-jms in the fuse container, but I downloaded service mix 4.5.1 which has camel 2.10.4 in it by default, and it seems to be working, hurray
          Hide
          Raúl Kripalani added a comment -

          Ok, never mind, I see the Camel version in the logs

          I don't know if FuseSource/Red Hat have backmerged some important JMS patches onto their 2.10.0.fuse-71-047 release.

          From the logs, it looks like they backmerged the initial support for concurrentConsumers on temp reply queues from Camel 2.10.3, but they didn't merge the latest fixes which actually make this feature useable (CAMEL-5865) in Camel 2.10.4. My reasoning is that I see 10 different temp queues being created for the same request queue, which is exactly what CAMEL-5865 resolves.

          Please give it a shot with Camel 2.10.4. You should be able to uninstall the current camel-jms component and install 2.10.4 with:

          install -s mvn:org.apache.camel/camel-jms/2.10.4
          

          Hopefully no more dependency upgrades will be necessary.

          Show
          Raúl Kripalani added a comment - Ok, never mind, I see the Camel version in the logs I don't know if FuseSource/Red Hat have backmerged some important JMS patches onto their 2.10.0.fuse-71-047 release. From the logs, it looks like they backmerged the initial support for concurrentConsumers on temp reply queues from Camel 2.10.3, but they didn't merge the latest fixes which actually make this feature useable ( CAMEL-5865 ) in Camel 2.10.4. My reasoning is that I see 10 different temp queues being created for the same request queue, which is exactly what CAMEL-5865 resolves. Please give it a shot with Camel 2.10.4. You should be able to uninstall the current camel-jms component and install 2.10.4 with: install -s mvn:org.apache.camel/camel-jms/2.10.4 Hopefully no more dependency upgrades will be necessary.
          Hide
          Raúl Kripalani added a comment -

          Thanks. Can you try with Camel JMS 2.10.4?

          Also, when you say you are on Camel 2.10, you mean Camel 2.10.0? What micro version exactly?

          Show
          Raúl Kripalani added a comment - Thanks. Can you try with Camel JMS 2.10.4? Also, when you say you are on Camel 2.10, you mean Camel 2.10.0? What micro version exactly?
          Alan Foster made changes -
          Attachment camel-6229.rar [ 12576306 ]
          Hide
          Alan Foster added a comment -

          Attached a sample project which allows you to reproduce this error.

          Note, I haven't been create a simple JUnit test that extends CamelBlueprintTestSupport, as I get a lot of strange exceptions - I'm not sure if this is a bug or not, but perhaps someone could shed light on it however?

          So currently the steps to reproduce this bug actually depend on it being deployed in a container, sorry

          1) install with : mvn install -Dmaven.test.skip=true
          2) Deploy to a container
          3) Run the junit test manually a couple of times. Sometimes the test will pass, but it also fails quite frequently

          Sorry again that I couldn't get it succesfully running as a JUnit test!

          Show
          Alan Foster added a comment - Attached a sample project which allows you to reproduce this error. Note, I haven't been create a simple JUnit test that extends CamelBlueprintTestSupport, as I get a lot of strange exceptions - I'm not sure if this is a bug or not, but perhaps someone could shed light on it however? So currently the steps to reproduce this bug actually depend on it being deployed in a container, sorry 1) install with : mvn install -Dmaven.test.skip=true 2) Deploy to a container 3) Run the junit test manually a couple of times. Sometimes the test will pass, but it also fails quite frequently Sorry again that I couldn't get it succesfully running as a JUnit test!
          Raúl Kripalani made changes -
          Field Original Value New Value
          Assignee Raul Kripalani [ raulvk ]
          Hide
          Raúl Kripalani added a comment -

          Two things:

          1. Could be related to CAMEL-5865. The InOut logic was a bit rough around the edges and was much improved in that ticket. Please try your code with Camel 2.10.4 and give us some feedback.
          2. Please post your route logic. In particular, I'd like to see your AMQ component configuration (JmsConfiguration) and the AMQ endpoint options, of both the producer and the consumer.

          Thanks.

          Show
          Raúl Kripalani added a comment - Two things: Could be related to CAMEL-5865 . The InOut logic was a bit rough around the edges and was much improved in that ticket. Please try your code with Camel 2.10.4 and give us some feedback. Please post your route logic. In particular, I'd like to see your AMQ component configuration (JmsConfiguration) and the AMQ endpoint options, of both the producer and the consumer. Thanks.
          Alan Foster created issue -

            People

            • Assignee:
              Raúl Kripalani
              Reporter:
              Alan Foster
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development