Geronimo
  1. Geronimo
  2. GERONIMO-2880

TransportDisposedIOException occurs when trying to close ActiveMQ queue

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.1
    • Fix Version/s: 2.0.3, 2.1
    • Component/s: ActiveMQ
    • Security Level: public (Regular issues)
    • Labels:
      None
    • Environment:

      Windows XP SP2

      Description

      I have discovered some problems with queues while running unittest in our own J2EE app.

      After sending a message on a queue, when we try to call the close() method on the queue, we get the following exception:


      org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#69) disposed.


      where the number after "localhost" is different every time.

      We do not experience this problem with topics. We are using ActiveMQ as part of an "embedded" configuration with Geronimo.

      I've done some debugging and the problem occurs at this line in the ActiveMQMessageProducer.close() method:


      this.session.asyncSendPacket(info.createRemoveCommand());


      The queue itself is disposed properly in the dispose() method that is called in the line before, but this sending of the asynchronous packet fails.

        Activity

        Hide
        Aman Nanner added a comment -

        Using queues with the embedded ActiveMQ server also produces a similar problem when trying to roll back a transaction:


        13:05:23,561 ERROR [Transaction] Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@7582d3; continuing with rollback
        javax.transaction.xa.XAException: Peer (vm://localhost#67) disposed.
        at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:609)
        at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:419)
        at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:127)
        at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
        at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:584)
        at org.apache.geronimo.transaction.manager.TransactionImpl.rollback(TransactionImpl.java:474)
        at org.apache.geronimo.transaction.manager.TransactionManagerImpl.rollback(TransactionManagerImpl.java:276)
        at org.apache.geronimo.transaction.manager.TransactionManagerImpl$$FastClassByCGLIB$$14ee5fe0.invoke(<generated>)
        at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
        at org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke(FastMethodInvoker.java:38)
        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:122)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:820)
        at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
        at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
        at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
        at org.apache.geronimo.transaction.manager.XidImporter$$EnhancerByCGLIB$$be1f3f2b.rollback(<generated>)
        at org.apache.geronimo.transaction.GeronimoUserTransaction.rollback(GeronimoUserTransaction.java:74)


        Again, this issue does not occur when using Topics.

        Show
        Aman Nanner added a comment - Using queues with the embedded ActiveMQ server also produces a similar problem when trying to roll back a transaction: 13:05:23,561 ERROR [Transaction] Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@7582d3; continuing with rollback javax.transaction.xa.XAException: Peer (vm://localhost#67) disposed. at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:609) at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:419) at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:127) at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78) at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:584) at org.apache.geronimo.transaction.manager.TransactionImpl.rollback(TransactionImpl.java:474) at org.apache.geronimo.transaction.manager.TransactionManagerImpl.rollback(TransactionManagerImpl.java:276) at org.apache.geronimo.transaction.manager.TransactionManagerImpl$$FastClassByCGLIB$$14ee5fe0.invoke(<generated>) at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53) at org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke(FastMethodInvoker.java:38) at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:122) at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:820) at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57) at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35) at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96) at org.apache.geronimo.transaction.manager.XidImporter$$EnhancerByCGLIB$$be1f3f2b.rollback(<generated>) at org.apache.geronimo.transaction.GeronimoUserTransaction.rollback(GeronimoUserTransaction.java:74) Again, this issue does not occur when using Topics.
        Hide
        Aman Nanner added a comment -

        Actually, I've now seen this issue with Topics now as well, specifically when calling the close() method on a TopicPublisher.

        Show
        Aman Nanner added a comment - Actually, I've now seen this issue with Topics now as well, specifically when calling the close() method on a TopicPublisher.
        Hide
        Aman Nanner added a comment -

        This issue seems to have resolved itself with an update to the latest ActiveMQ 4.1 snapshot.

        Show
        Aman Nanner added a comment - This issue seems to have resolved itself with an update to the latest ActiveMQ 4.1 snapshot.
        Hide
        Aman Nanner added a comment -

        This issue is now occurring in Geronimo 2.0.1

        Show
        Aman Nanner added a comment - This issue is now occurring in Geronimo 2.0.1
        Hide
        Aman Nanner added a comment -

        This issue is occuring when we try to close a Queue.

        10:42:34,957 ERROR [Transaction] Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@70e59f; continuing with rollback
        javax.transaction.xa.XAException: Peer (vm://localhost#79) disposed.
        at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:619)
        at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:420)
        at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:127)
        at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
        at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
        at org.apache.geronimo.transaction.manager.TransactionImpl.rollback(TransactionImpl.java:475)
        at org.apache.geronimo.transaction.manager.TransactionManagerImpl.rollback(TransactionManagerImpl.java:250)
        at org.apache.geronimo.transaction.manager.TransactionManagerImpl$$FastClassByCGLIB$$14ee5fe0.invoke(<generated>)
        at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
        at org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke(FastMethodInvoker.java:38)
        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:830)
        at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
        at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
        at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
        at org.apache.geronimo.transaction.manager.XidImporter$$EnhancerByCGLIB$$abb1517c.rollback(<generated>)
        at org.apache.geronimo.transaction.GeronimoUserTransaction.rollback(GeronimoUserTransaction.java:74)
        .................
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.apache.geronimo.tomcat.valve.DefaultSubjectValve.invoke(DefaultSubjectValve.java:56)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
        at org.apache.geronimo.tomcat.GeronimoStandardContext$SystemMethodValve.invoke(GeronimoStandardContext.java:353)
        at org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke(GeronimoBeforeAfterValve.java:47)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:393)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:595)
        Caused by: javax.jms.JMSException: Peer (vm://localhost#79) disposed.
        at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:58)
        at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1185)
        at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:409)
        ... 62 more
        Caused by: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#79) disposed.
        at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:86)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47)
        at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
        at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
        at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1175)
        ... 63 more

        Show
        Aman Nanner added a comment - This issue is occuring when we try to close a Queue. 10:42:34,957 ERROR [Transaction] Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@70e59f; continuing with rollback javax.transaction.xa.XAException: Peer (vm://localhost#79) disposed. at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:619) at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:420) at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:127) at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78) at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581) at org.apache.geronimo.transaction.manager.TransactionImpl.rollback(TransactionImpl.java:475) at org.apache.geronimo.transaction.manager.TransactionManagerImpl.rollback(TransactionManagerImpl.java:250) at org.apache.geronimo.transaction.manager.TransactionManagerImpl$$FastClassByCGLIB$$14ee5fe0.invoke(<generated>) at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53) at org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke(FastMethodInvoker.java:38) at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124) at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:830) at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57) at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35) at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96) at org.apache.geronimo.transaction.manager.XidImporter$$EnhancerByCGLIB$$abb1517c.rollback(<generated>) at org.apache.geronimo.transaction.GeronimoUserTransaction.rollback(GeronimoUserTransaction.java:74) ................. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.apache.geronimo.tomcat.valve.DefaultSubjectValve.invoke(DefaultSubjectValve.java:56) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) at org.apache.geronimo.tomcat.GeronimoStandardContext$SystemMethodValve.invoke(GeronimoStandardContext.java:353) at org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke(GeronimoBeforeAfterValve.java:47) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:393) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:595) Caused by: javax.jms.JMSException: Peer (vm://localhost#79) disposed. at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:58) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1185) at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:409) ... 62 more Caused by: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#79) disposed. at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:86) at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47) at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69) at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1175) ... 63 more
        Hide
        Aman Nanner added a comment -

        Ok, it turns out that the peer was disposed because of an exception during the transport process. The real problem was the following IOException:

        java.io.IOException: Cannot set AUTOCOMMIT ON when in an XA connection.

        This problem caused the peer to be disposed which resulted in the error discussed in this JIRA issue. However, this source problem was not logged as an ERROR to the console. It seems that the error is only logged in DEBUG mode in the following code in the org.apache.activemq.broker.TransportConnection class:

            public void serviceTransportException(IOException e) {
                if( !disposed.get() ) {
                    transportException.set(e); 
                    if( transportLog.isDebugEnabled() )
                        transportLog.debug("Transport failed: "+e,e);
                    ServiceSupport.dispose(this);
                }
            }
        

        I would think that this should probably be logged at a higher debug level, such as WARN or ERROR.

        Show
        Aman Nanner added a comment - Ok, it turns out that the peer was disposed because of an exception during the transport process. The real problem was the following IOException: java.io.IOException: Cannot set AUTOCOMMIT ON when in an XA connection. This problem caused the peer to be disposed which resulted in the error discussed in this JIRA issue. However, this source problem was not logged as an ERROR to the console. It seems that the error is only logged in DEBUG mode in the following code in the org.apache.activemq.broker.TransportConnection class: public void serviceTransportException(IOException e) { if ( !disposed.get() ) { transportException.set(e); if ( transportLog.isDebugEnabled() ) transportLog.debug( "Transport failed: " +e,e); ServiceSupport.dispose( this ); } } I would think that this should probably be logged at a higher debug level, such as WARN or ERROR.
        Hide
        Anish Pathadan added a comment -

        Hi Aman,
        I could successfully send messages from a standalone client as well as from a servlet.There was no exception while closing the session or connection. Can you provide a test case for the issue.

        Thanks,
        Anish Pathadan

        Show
        Anish Pathadan added a comment - Hi Aman, I could successfully send messages from a standalone client as well as from a servlet.There was no exception while closing the session or connection. Can you provide a test case for the issue. Thanks, Anish Pathadan
        Hide
        Aman Nanner added a comment -

        Hi Anish,

        As per my last comment, I had discovered that the exception was being generated because the connection was already closed by ActiveMQ due to an error that occurred during the sending of the last message. This error message did not appear in my log because my logging threshold was set above DEBUG, so I had suggested that errors that occur during the sending of message should perhaps be logged at a higher threshold level (such as WARN or ERROR).

        Also, the source problem was this:

        java.io.IOException: Cannot set AUTOCOMMIT ON when in an XA connection.

        Does anybody know how to set AUTOCOMMIT to off for the JMS resource adapter?

        Show
        Aman Nanner added a comment - Hi Anish, As per my last comment, I had discovered that the exception was being generated because the connection was already closed by ActiveMQ due to an error that occurred during the sending of the last message. This error message did not appear in my log because my logging threshold was set above DEBUG, so I had suggested that errors that occur during the sending of message should perhaps be logged at a higher threshold level (such as WARN or ERROR). Also, the source problem was this: java.io.IOException: Cannot set AUTOCOMMIT ON when in an XA connection. Does anybody know how to set AUTOCOMMIT to off for the JMS resource adapter?
        Hide
        Anish Pathadan added a comment -

        Hi All,
        The issue is due to geronimo using XA datasource (SystemDatasource) for activemq persistence. I have discussed this issue in activemq forum and I got the recommendation to use a non XA datasource for activemq persistence. The reason is activemq itself is managing the transaction for broker persistence and it doesn't make sense to use XA datasource as there is no other Transaction Manager involved in it.
        I have modified the activemq-broker plan to use NoTxDatasource and I got the issue resolved.I am attaching a patch also.I verified the issue in AG 2.0.1 release.

        Thanks,
        Anish Pathadan

        Show
        Anish Pathadan added a comment - Hi All, The issue is due to geronimo using XA datasource (SystemDatasource) for activemq persistence. I have discussed this issue in activemq forum and I got the recommendation to use a non XA datasource for activemq persistence. The reason is activemq itself is managing the transaction for broker persistence and it doesn't make sense to use XA datasource as there is no other Transaction Manager involved in it. I have modified the activemq-broker plan to use NoTxDatasource and I got the issue resolved.I am attaching a patch also.I verified the issue in AG 2.0.1 release. Thanks, Anish Pathadan
        Hide
        Vamsavardhana Reddy added a comment -

        Completed: At revision: 589532
        o Thanks to Anish for providing the patch
        o Use a non XA datasource for activemq persistence
        **: This commit can use a review.

        Show
        Vamsavardhana Reddy added a comment - Completed: At revision: 589532 o Thanks to Anish for providing the patch o Use a non XA datasource for activemq persistence **: This commit can use a review.
        Hide
        David Jencks added a comment -

        This fix looks completely correct to me and I'm only embarrassed that I didn't use the right datasource in the first place.

        Show
        David Jencks added a comment - This fix looks completely correct to me and I'm only embarrassed that I didn't use the right datasource in the first place.

          People

          • Assignee:
            Vamsavardhana Reddy
            Reporter:
            Aman Nanner
          • Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development