ActiveMQ C++ Client
  1. ActiveMQ C++ Client
  2. AMQCPP-388

AprPool::getAprPool() returns NULL, causing access violation and application crash

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.2.3
    • Fix Version/s: 3.2.3
    • Component/s: Decaf
    • Labels:
      None
    • Environment:

      Windows xp service pack 3, ActiveMQ broker 5.3.1, apr 1.4.2, apr-util 1.3.9, apr iconv 1.2.1

      Description

      Our application that uses activemq c++ client lib crashed with the following dump:

      ----------------------------------------------------------------------------------------------------------------------
      Thread 87 - System ID 3780

      Function Arg 1 Arg 2 Arg 3 Source
      libapr_1!apr_pvsprintf+8 00000000 0642a188 180eeb94
      activemq_cppu!decaf::lang::Exception::buildMessage+71 0642a188 180eeb74 180eee2c
      activemq_cppu!decaf::lang::exceptions::RuntimeException::RuntimeException+4d 180efeec 0642a160 00000097
      activemq_cppu!decaf::lang::ThreadProperties::runCallback+125 180eee2c 180efee0 00000001
      msvcr80!CatchIt+5c 00000000 00000000 00000000

      LIBAPR_1!APR_PVSPRINTF+8In scotapp.dmp the assembly instruction at libapr_1!apr_pvsprintf+8 in C:\scot\dll\libapr-1.dll has caused an access violation exception (0xC0000005) when trying to read from memory location 0x0000002c on thread 87
      --------------------------------------------------------------------------------------------------------------------

      On the call stack, we saw that following function from class decaf::lang::Exception was called:
      void Exception::buildMessage( const char* format, va_list& vargs )

      { // Allocate buffer with a guess of it's size AprPool pool; // Allocate a buffer of the specified size. char* buffer = apr_pvsprintf( pool.getAprPool(), format, vargs ); // Guessed size was enough. Assign the string. message.assign( buffer, strlen( buffer ) ); }

      The first parameter passed into apr_pvsprintf was NULL, causing the crash. Could you please take a look and see if there is any bug in the activemqcpp code that could cause the problem. Thanks!

      1. AMQCPP-388-Patch.txt
        8 kB
        Timothy Bish
      2. AMQCPP-388-Patch.txt
        3 kB
        Timothy Bish
      3. BrokerMonitor.zip
        9.65 MB
        Helen Huang

        Activity

        Hide
        Timothy Bish added a comment -

        Recommend you switch to v3.4.1 as there are many fixes in there. Otherwise we'd need a unit test that shows the problem to really narrow it down.

        Show
        Timothy Bish added a comment - Recommend you switch to v3.4.1 as there are many fixes in there. Otherwise we'd need a unit test that shows the problem to really narrow it down.
        Hide
        Helen Huang added a comment -

        We do not have a unit test that could reproduce the problem. So we are thinking of switching to v3.4.1

        Could you please tell us the versions of the apr libs that v3.4.1 is using?

        Thanks!

        Show
        Helen Huang added a comment - We do not have a unit test that could reproduce the problem. So we are thinking of switching to v3.4.1 Could you please tell us the versions of the apr libs that v3.4.1 is using? Thanks!
        Hide
        Timothy Bish added a comment -

        v3.4.1 is compatible with the same APR libs as previous versions, there should be no need to change versions.

        Show
        Timothy Bish added a comment - v3.4.1 is compatible with the same APR libs as previous versions, there should be no need to change versions.
        Hide
        Helen Huang added a comment -

        I got several compiler errors when I was trying to build our app with v3.4.1

        One error is that "decaf/util/concurrent/TaskListener.h" seems to be missing. In our app, we are using ThreadPool, Task, and TaskListener that are available in v3.2.3. I am wondering if they are supported in v3.4.1. If not, do you provide similar utilities in the new version? Also, it will be very much helpful if you can point me to some examples that use the new utilities. Thanks a lot for your help!

        Show
        Helen Huang added a comment - I got several compiler errors when I was trying to build our app with v3.4.1 One error is that "decaf/util/concurrent/TaskListener.h" seems to be missing. In our app, we are using ThreadPool, Task, and TaskListener that are available in v3.2.3. I am wondering if they are supported in v3.4.1. If not, do you provide similar utilities in the new version? Also, it will be very much helpful if you can point me to some examples that use the new utilities. Thanks a lot for your help!
        Hide
        Timothy Bish added a comment -

        The ThreadPool class was removed since it wasn't used by any of the ActiveMQ-CPP code and had some limitations as well as a few bugs that weren't so great. Depending on what you need there are a couple utility classes that may help here, activemq::thread contains a few different Task runners that can be used to run jobs async. Also the decaf::util::concurrent namespace contained a ThreadPoolExecutor that takes Runnable instance to execute using a pool. The one is v3.4.1 is a bit primitive but works about as well as the old ThreadPool class.

        Show
        Timothy Bish added a comment - The ThreadPool class was removed since it wasn't used by any of the ActiveMQ-CPP code and had some limitations as well as a few bugs that weren't so great. Depending on what you need there are a couple utility classes that may help here, activemq::thread contains a few different Task runners that can be used to run jobs async. Also the decaf::util::concurrent namespace contained a ThreadPoolExecutor that takes Runnable instance to execute using a pool. The one is v3.4.1 is a bit primitive but works about as well as the old ThreadPool class.
        Hide
        Timothy Bish added a comment -

        Since the code is Apache licensed you are always free to extract the old ThreadPool class and TaskListener bits into your own codebase.

        Show
        Timothy Bish added a comment - Since the code is Apache licensed you are always free to extract the old ThreadPool class and TaskListener bits into your own codebase.
        Hide
        Helen Huang added a comment -

        I was trying to use the ThreadPoolExecutor, but could not resolve a link error. Could you please take a look?

        Receiver.obj : error LNK2001: unresolved external symbol "public: static class decaf::util::concurrent::TimeUnit const decaf::util::concurrent::TimeUnit::MILLISECONDS" (?MILLISECONDS@TimeUnit@concurrent@util@decaf@@2V1234@B)
        E:\TopCoderBackup\BrokerMonitor\DebugUnicode\BrokerMonitor.exe : fatal error LNK1120: 1 unresolved externals

        Attached please find a mini project that can be used to show this error.

        Show
        Helen Huang added a comment - I was trying to use the ThreadPoolExecutor, but could not resolve a link error. Could you please take a look? Receiver.obj : error LNK2001: unresolved external symbol "public: static class decaf::util::concurrent::TimeUnit const decaf::util::concurrent::TimeUnit::MILLISECONDS" (?MILLISECONDS@TimeUnit@concurrent@util@decaf@@2V1234@B) E:\TopCoderBackup\BrokerMonitor\DebugUnicode\BrokerMonitor.exe : fatal error LNK1120: 1 unresolved externals Attached please find a mini project that can be used to show this error.
        Hide
        Helen Huang added a comment -

        I just resolved the link issue by adding the preprocessor definition DECAF_DLL to the project. Thanks!

        Show
        Helen Huang added a comment - I just resolved the link issue by adding the preprocessor definition DECAF_DLL to the project. Thanks!
        Hide
        Helen Huang added a comment -

        I have updated my code to use v3.4.1. However, sometimes the app crashes because access violations in CMS. I found a couple of places that caused the issue. Could you please take a look?

        Sometimes, this->config on the following line seems to be invalid and points to garbage
        std::auto_ptr< Iterator<ActiveMQProducer*> > producerIter( this->config->producers.iterator() );

        Sometimes, this->transaction on the the following line seems to be an invalid pointer
        if( this->transaction->isInTransaction() )

        Show
        Helen Huang added a comment - I have updated my code to use v3.4.1. However, sometimes the app crashes because access violations in CMS. I found a couple of places that caused the issue. Could you please take a look? Sometimes, this->config on the following line seems to be invalid and points to garbage std::auto_ptr< Iterator<ActiveMQProducer*> > producerIter( this->config->producers.iterator() ); Sometimes, this->transaction on the the following line seems to be an invalid pointer if( this->transaction->isInTransaction() )
        Hide
        Timothy Bish added a comment -

        can you provide info on source and line numbers for those code bits? If you can reproduce in a test case its also helpful.

        Show
        Timothy Bish added a comment - can you provide info on source and line numbers for those code bits? If you can reproduce in a test case its also helpful.
        Hide
        Helen Huang added a comment -

        The crash happened in ActiveMQSession::dispose()

        line 357
        std::auto_ptr< Iterator<ActiveMQProducer*> > producerIter( this->config->producers.iterator() );

        line 335
        if( this->transaction->isInTransaction() )

        Show
        Helen Huang added a comment - The crash happened in ActiveMQSession::dispose() line 357 std::auto_ptr< Iterator<ActiveMQProducer*> > producerIter( this->config->producers.iterator() ); line 335 if( this->transaction->isInTransaction() )
        Hide
        Timothy Bish added a comment -

        My first guess would be that there are seperate threads closing the Session along with its associated Producers. This could result in this sort of issue. The Session and its associated resources in CMS adhere to the same general rules as those in JMS when it comes to threading. Its generally discouraged to use a session resource in a thread other than the one it and its session where created in.

        Show
        Timothy Bish added a comment - My first guess would be that there are seperate threads closing the Session along with its associated Producers. This could result in this sort of issue. The Session and its associated resources in CMS adhere to the same general rules as those in JMS when it comes to threading. Its generally discouraged to use a session resource in a thread other than the one it and its session where created in.
        Hide
        Helen Huang added a comment -

        Thanks for looking at this issue. We do not directly use the ActiveMQSession class. We use the CmsTemplate class instead. Attached please find a test program that can be used to reproduce the problem.

        Steps:
        (1) start the BrokerMoniter application
        (2) while the BrokerMoniter app is running, restart the message broker

        The issue is hard to reproduce. I had to repeat (2) a number of times to create the crash. Also, it is easier to recreate the problem when running the program from VC2005 IDE.

        Show
        Helen Huang added a comment - Thanks for looking at this issue. We do not directly use the ActiveMQSession class. We use the CmsTemplate class instead. Attached please find a test program that can be used to reproduce the problem. Steps: (1) start the BrokerMoniter application (2) while the BrokerMoniter app is running, restart the message broker The issue is hard to reproduce. I had to repeat (2) a number of times to create the crash. Also, it is easier to recreate the problem when running the program from VC2005 IDE.
        Hide
        Helen Huang added a comment -

        I have updated the test program in the attachment today. Please get the latest. Thanks!

        Also, there is the call stack of the crash:

        > activemq-cppud.dll!activemq::core::ActiveMQSession::dispose() Line 335 + 0x26 bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::cleanup() Line 466 + 0x8 bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex=

        {...}) Line 783 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 56 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 49 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 56 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 49 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 56 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 49 C++
        activemq-cppud.dll!activemq::transport::inactivity::InactivityMonitor::onException(const decaf::lang::Exception & ex={...}

        ) Line 314 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex=

        {...}) Line 56 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex={...}

        ) Line 49 C++
        activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex=

        {...}

        ) Line 72 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++
        activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x01332730) Line 137 + 0x13 bytes C++
        activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x01332730) Line 210 + 0x9 bytes C++
        msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
        msvcr80d.dll!_threadstartex(void * ptd=0x01315a38) Line 331 C

        Show
        Helen Huang added a comment - I have updated the test program in the attachment today. Please get the latest. Thanks! Also, there is the call stack of the crash: > activemq-cppud.dll!activemq::core::ActiveMQSession::dispose() Line 335 + 0x26 bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::cleanup() Line 466 + 0x8 bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex= {...}) Line 783 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 56 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 49 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 56 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 49 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 56 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 49 C++ activemq-cppud.dll!activemq::transport::inactivity::InactivityMonitor::onException(const decaf::lang::Exception & ex={...} ) Line 314 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex= {...}) Line 56 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex={...} ) Line 49 C++ activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex= {...} ) Line 72 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++ activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x01332730) Line 137 + 0x13 bytes C++ activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x01332730) Line 210 + 0x9 bytes C++ msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C msvcr80d.dll!_threadstartex(void * ptd=0x01315a38) Line 331 C
        Hide
        Helen Huang added a comment -

        Updated the priority of this issue to "Critical" because we see it in production. We hope you can resolve it soon. Thanks!

        Show
        Helen Huang added a comment - Updated the priority of this issue to "Critical" because we see it in production. We hope you can resolve it soon. Thanks!
        Hide
        Helen Huang added a comment - - edited

        I think adding a call to synchronize the access to "this->config->activeSessions" in ActiveMQConnection.cpp::cleanup() might help. (See details below)

        I know "this->config->activeSessions" has a mutex to synchronize the addition and removal of a session from that array. But multiple threads can still use iterators to access/despose the sessions directly. We can prevent this to happen by calling
        "this->config->activeSessions.lock();" before using the iterator.

        void ActiveMQConnection::cleanup() {

        try{

        // Get the complete list of active sessions.

        this->config->activeSessions.lock();

        std::auto_ptr< Iterator<ActiveMQSession*> > iter( this->config->activeSessions.iterator() );

        // Dispose of all the Session resources we know are still open.
        while( iter->hasNext() ) {
        ActiveMQSession* session = iter->next();
        try

        { session->dispose(); }

        catch( cms::CMSException& ex )

        { /* Absorb */ }

        }

        this->config->activeSessions.unlock();

        if( this->config->isConnectionInfoSentToBroker ) {
        if( !transportFailed.get() && !closing.get() )

        { this->syncRequest( this->config->connectionInfo->createRemoveCommand() ); }

        this->config->isConnectionInfoSentToBroker = false;
        }

        if( this->config->userSpecifiedClientID )

        { this->config->connectionInfo->setClientId(""); this->config->userSpecifiedClientID = false; }

        this->config->clientIDSet = false;
        this->started.set( false );
        }
        AMQ_CATCH_ALL_THROW_CMSEXCEPTION()
        }

        Show
        Helen Huang added a comment - - edited I think adding a call to synchronize the access to "this->config->activeSessions" in ActiveMQConnection.cpp::cleanup() might help. (See details below) I know "this->config->activeSessions" has a mutex to synchronize the addition and removal of a session from that array. But multiple threads can still use iterators to access/despose the sessions directly. We can prevent this to happen by calling "this->config->activeSessions.lock();" before using the iterator. void ActiveMQConnection::cleanup() { try{ // Get the complete list of active sessions. this->config->activeSessions.lock(); std::auto_ptr< Iterator<ActiveMQSession*> > iter( this->config->activeSessions.iterator() ); // Dispose of all the Session resources we know are still open. while( iter->hasNext() ) { ActiveMQSession* session = iter->next(); try { session->dispose(); } catch( cms::CMSException& ex ) { /* Absorb */ } } this->config->activeSessions.unlock(); if( this->config->isConnectionInfoSentToBroker ) { if( !transportFailed.get() && !closing.get() ) { this->syncRequest( this->config->connectionInfo->createRemoveCommand() ); } this->config->isConnectionInfoSentToBroker = false; } if( this->config->userSpecifiedClientID ) { this->config->connectionInfo->setClientId(""); this->config->userSpecifiedClientID = false; } this->config->clientIDSet = false; this->started.set( false ); } AMQ_CATCH_ALL_THROW_CMSEXCEPTION() }
        Hide
        Helen Huang added a comment - - edited

        I tried to add the lock that I have suggested to ActiveMQConnection::cleanup(), repeated the test a few times, and did not get the same crash as before. So that might have worked. However, I got another crash in ActiveMQConnection::onException() after the cleanup() call. The error message was: R6025 -pure virtual function call. Could you please take a look? Thanks!

        Call stack:
        activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex=

        {...}) Line 787 C++
        > activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 58 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 51 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 58 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 51 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 58 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 51 C++
        activemq-cppud.dll!activemq::transport::inactivity::InactivityMonitor::onException(const decaf::lang::Exception & ex={...}

        ) Line 314 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex=

        {...}) Line 58 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex={...}

        ) Line 51 C++
        activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex=

        {...}

        ) Line 72 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++
        activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x012ea1d8) Line 137 + 0x13 bytes C++
        activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x012ea1d8) Line 210 + 0x9 bytes C++
        msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
        msvcr80d.dll!_threadstartex(void * ptd=0x012e9f88) Line 331 C
        kernel32.dll!7c80b713()

        It looks like the connection used by the above call stack can be disconnected by a second thread. The following is the call stack of that thread:

        > activemq-cppud.dll!activemq::core::ActiveMQConnection::disconnect(__int64 lastDeliveredSequenceId=0) Line 542 C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::close() Line 448 C++
        activemq-cppud.dll!activemq::cmsutil::ResourceLifecycleManager::destroy() Line 81 + 0x11 bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsAccessor::destroy() Line 131 C++
        activemq-cppud.dll!activemq::cmsutil::CmsDestinationAccessor::destroy() Line 58 C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::destroy() Line 211 + 0xb bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::ProducerExecutor::doInCms(cms::Session * session=0x01213518) Line 527 + 0x1a bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::execute(activemq::cmsutil::SessionCallback * action=0x0120fc34) Line 446 C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::execute(activemq::cmsutil::ProducerCallback * action=0x0120fcbc) Line 465 C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::send(activemq::cmsutil::MessageCreator * messageCreator=0x00b7e144) Line 546 C++
        BrokerMonitor.exe!BrokerMonitor::run() Line 123 + 0x37 bytes C++
        activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x00b7e380) Line 137 + 0x13 bytes C++
        activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x00b7e380) Line 210 + 0x9 bytes C++
        msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
        msvcr80d.dll!_threadstartex(void * ptd=0x00b7efd8) Line 331 C

        Show
        Helen Huang added a comment - - edited I tried to add the lock that I have suggested to ActiveMQConnection::cleanup(), repeated the test a few times, and did not get the same crash as before. So that might have worked. However, I got another crash in ActiveMQConnection::onException() after the cleanup() call. The error message was: R6025 -pure virtual function call. Could you please take a look? Thanks! Call stack: activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex= {...}) Line 787 C++ > activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 58 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 51 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 58 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 51 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 58 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 51 C++ activemq-cppud.dll!activemq::transport::inactivity::InactivityMonitor::onException(const decaf::lang::Exception & ex={...} ) Line 314 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex= {...}) Line 58 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex={...} ) Line 51 C++ activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex= {...} ) Line 72 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++ activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x012ea1d8) Line 137 + 0x13 bytes C++ activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x012ea1d8) Line 210 + 0x9 bytes C++ msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C msvcr80d.dll!_threadstartex(void * ptd=0x012e9f88) Line 331 C kernel32.dll!7c80b713() It looks like the connection used by the above call stack can be disconnected by a second thread. The following is the call stack of that thread: > activemq-cppud.dll!activemq::core::ActiveMQConnection::disconnect(__int64 lastDeliveredSequenceId=0) Line 542 C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::close() Line 448 C++ activemq-cppud.dll!activemq::cmsutil::ResourceLifecycleManager::destroy() Line 81 + 0x11 bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsAccessor::destroy() Line 131 C++ activemq-cppud.dll!activemq::cmsutil::CmsDestinationAccessor::destroy() Line 58 C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::destroy() Line 211 + 0xb bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::ProducerExecutor::doInCms(cms::Session * session=0x01213518) Line 527 + 0x1a bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::execute(activemq::cmsutil::SessionCallback * action=0x0120fc34) Line 446 C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::execute(activemq::cmsutil::ProducerCallback * action=0x0120fcbc) Line 465 C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::send(activemq::cmsutil::MessageCreator * messageCreator=0x00b7e144) Line 546 C++ BrokerMonitor.exe!BrokerMonitor::run() Line 123 + 0x37 bytes C++ activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x00b7e380) Line 137 + 0x13 bytes C++ activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x00b7e380) Line 210 + 0x9 bytes C++ msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C msvcr80d.dll!_threadstartex(void * ptd=0x00b7efd8) Line 331 C
        Hide
        Helen Huang added a comment -

        I am thinking it might help to add a mutex to the Transport class to synchronize the access to its listener variable. TransportFilter::onException() and ActiveMQConnection::disconnect() would both use this mutex. Not sure if this makes sense or not to you or not... What do you think?

        Show
        Helen Huang added a comment - I am thinking it might help to add a mutex to the Transport class to synchronize the access to its listener variable. TransportFilter::onException() and ActiveMQConnection::disconnect() would both use this mutex. Not sure if this makes sense or not to you or not... What do you think?
        Hide
        Timothy Bish added a comment -

        We need to be extremely careful when adding additional locking due to wide range of usecases its easy to create a deadlock. Will have to analyze the stack traces when I have time and determine the best course of action. Not sure in CMSTemplate where the additional threads are so can't say why the session close and connection close are contending.

        Show
        Timothy Bish added a comment - We need to be extremely careful when adding additional locking due to wide range of usecases its easy to create a deadlock. Will have to analyze the stack traces when I have time and determine the best course of action. Not sure in CMSTemplate where the additional threads are so can't say why the session close and connection close are contending.
        Hide
        Helen Huang added a comment -


        Thanks a lot for looking at this problem. We would appreciate an update on this issue whenever possible.

        Show
        Helen Huang added a comment - Thanks a lot for looking at this problem. We would appreciate an update on this issue whenever possible.
        Hide
        Helen Huang added a comment -

        While you are working on resolving this problem, we are wondering if there is any way to avoid it. Can we turn off the InactivityMonitor? Is there any harm if we turn it off? Thanks!

        Show
        Helen Huang added a comment - While you are working on resolving this problem, we are wondering if there is any way to avoid it. Can we turn off the InactivityMonitor? Is there any harm if we turn it off? Thanks!
        Hide
        Helen Huang added a comment -

        I tried to turn off the inactivity monitor by using the following url, but it did not work. The InActivityMonitor class was still placed on the call stack leading to the same crash. Could you please take a look and see if I did it wrong. Much thanks for your help!

        The new url: (I changed the url on line 58 in main.cpp of the attached BrokerMonitor program)

        ActiveMQConnectionFactory* connectionFactory=new ActiveMQConnectionFactory("tcp://127.0.0.1:61616?connection.sendTimeout=2000&wireFormat.maxInactivityDuration=0");

        Note: We use CmsTemplate to send and receiver messages. This connection factory is used to create the CmsTemplate instances.

        Call stack at crash:

        activemq-cppud.dll!activemq::core::ActiveMQSession::dispose() Line 357 + 0x9 bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::cleanup() Line 466 + 0x8 bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex=

        {...}) Line 783 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        > activemq-cppud.dll!activemq::transport::inactivity::InactivityMonitor::onException(const decaf::lang::Exception & ex={...}

        ) Line 314 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex=

        {...}) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex={...}

        ) Line 47 C++
        activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex=

        {...}

        ) Line 72 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++
        activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x01401000) Line 137 + 0x13 bytes C++
        activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x01401000) Line 210 + 0x9 bytes C++
        msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
        msvcr80d.dll!_threadstartex(void * ptd=0x01400db0) Line 331 C

        Show
        Helen Huang added a comment - I tried to turn off the inactivity monitor by using the following url, but it did not work. The InActivityMonitor class was still placed on the call stack leading to the same crash. Could you please take a look and see if I did it wrong. Much thanks for your help! The new url: (I changed the url on line 58 in main.cpp of the attached BrokerMonitor program) ActiveMQConnectionFactory* connectionFactory=new ActiveMQConnectionFactory("tcp://127.0.0.1:61616?connection.sendTimeout=2000&wireFormat.maxInactivityDuration=0"); Note: We use CmsTemplate to send and receiver messages. This connection factory is used to create the CmsTemplate instances. Call stack at crash: activemq-cppud.dll!activemq::core::ActiveMQSession::dispose() Line 357 + 0x9 bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::cleanup() Line 466 + 0x8 bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex= {...}) Line 783 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ > activemq-cppud.dll!activemq::transport::inactivity::InactivityMonitor::onException(const decaf::lang::Exception & ex={...} ) Line 314 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex= {...}) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex={...} ) Line 47 C++ activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex= {...} ) Line 72 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++ activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x01401000) Line 137 + 0x13 bytes C++ activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x01401000) Line 210 + 0x9 bytes C++ msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C msvcr80d.dll!_threadstartex(void * ptd=0x01400db0) Line 331 C
        Hide
        Timothy Bish added a comment -

        to completely disable the inactivity monitor you can set transport.useInactivityMonitor=false on the connection URI.

        Show
        Timothy Bish added a comment - to completely disable the inactivity monitor you can set transport.useInactivityMonitor=false on the connection URI.
        Hide
        Helen Huang added a comment - - edited

        Hi Timothy,

        Thanks for the help on disabling the inactivity monitor. Now the URI I use is "tcp://127.0.0.1:61616?connection.sendTimeout=2000&transport.useInactivityMonitor=false". However, we found another crash when we restarted the broker. The following is the call stack of the crashing thread. I am wondering if there is any walkaround for it. Thanks!

        activemq-cppud.dll!activemq::core::ActiveMQSession::dispose() Line 357 + 0x9 bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::cleanup() Line 466 + 0x8 bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex=

        {...}) Line 783 C++
        > activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex={...}

        ) Line 72 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++
        activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x01300870) Line 137 + 0x13 bytes C++
        activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x01300870) Line 210 + 0x9 bytes C++
        msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
        msvcr80d.dll!_threadstartex(void * ptd=0x01301318) Line 331 C

        Show
        Helen Huang added a comment - - edited Hi Timothy, Thanks for the help on disabling the inactivity monitor. Now the URI I use is "tcp://127.0.0.1:61616?connection.sendTimeout=2000&transport.useInactivityMonitor=false". However, we found another crash when we restarted the broker. The following is the call stack of the crashing thread. I am wondering if there is any walkaround for it. Thanks! activemq-cppud.dll!activemq::core::ActiveMQSession::dispose() Line 357 + 0x9 bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::cleanup() Line 466 + 0x8 bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex= {...}) Line 783 C++ > activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex={...} ) Line 72 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++ activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x01300870) Line 137 + 0x13 bytes C++ activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x01300870) Line 210 + 0x9 bytes C++ msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C msvcr80d.dll!_threadstartex(void * ptd=0x01301318) Line 331 C
        Hide
        Timothy Bish added a comment -

        Possible fix for your segv issues.

        Show
        Timothy Bish added a comment - Possible fix for your segv issues.
        Hide
        Timothy Bish added a comment -

        Added some synchronization to the access of Active Session in ActiveMQConnection, you can test and see if this has any effect on your crash problems. I'm not able to reproduce locally.

        Show
        Timothy Bish added a comment - Added some synchronization to the access of Active Session in ActiveMQConnection, you can test and see if this has any effect on your crash problems. I'm not able to reproduce locally.
        Hide
        Helen Huang added a comment -

        Thanks Timothy! I will try it soon and let you know how it goes.

        Show
        Helen Huang added a comment - Thanks Timothy! I will try it soon and let you know how it goes.
        Hide
        Helen Huang added a comment - - edited

        Hi Timothy, I have tested the new ActiveMQConnection.cpp, but got a "pure virtual function call" exception from ActiveMQConnection::CleanUp(). From the call stack, I found TransportFilter::fire was called before ActiveMQConnection::CleanUp().

        msvcr80d.dll!_NMSG_WRITE(int rterrnum=25) Line 198 C
        msvcr80d.dll!_purecall() Line 54 + 0x7 bytes C
        activemq-cppud.dll!activemq::core::ActiveMQConnection::cleanup() Line 469 + 0x49 bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex=

        {...}) Line 793 C++
        > activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...}

        ) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex=

        {...}) Line 47 C++
        activemq-cppud.dll!activemq::transport::inactivity::InactivityMonitor::onException(const decaf::lang::Exception & ex={...}

        ) Line 314 C++
        activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex=

        {...}) Line 54 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex={...}

        ) Line 47 C++
        activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex=

        {...}

        ) Line 72 + 0x17 bytes C++
        activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++
        activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x0131b718) Line 137 + 0x13 bytes C++
        activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x0131b718) Line 210 + 0x9 bytes C++
        msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C
        msvcr80d.dll!_threadstartex(void * ptd=0x01333ba8) Line 331 C

        In function TransportFilter::fire(), the listener pointer was NULL at the crash. I guess some other thread might have deleted that listener/ActiveMQConnection while this thread is in ActiveMQConnection::cleanup().

        void TransportFilter::fire( const decaf::lang::Exception& ex ) {

        if( listener != NULL ){
        try

        { listener->onException( ex ); }

        catch( ... ){}
        }
        }

        I have put a break point in the destructor of ActiveMQConnection, and found these threads could delete it when the broker goes down:

        Call Stack 1:
        activemq-cppud.dll!activemq::core::ActiveMQConnection::~ActiveMQConnection() Line 242 C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::`vbase destructor'() + 0xf bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::`vector deleting destructor'() + 0x4d bytes C++
        activemq-cppud.dll!activemq::cmsutil::ResourceLifecycleManager::destroy() Line 135 + 0x20 bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsAccessor::destroy() Line 131 C++
        activemq-cppud.dll!activemq::cmsutil::CmsDestinationAccessor::destroy() Line 58 C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::destroy() Line 211 + 0xb bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::doReceive(cms::MessageConsumer * consumer=0x01408d60) Line 632 + 0x1f bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::ReceiveExecutor::doInCms(cms::Session * session=0x01405760) Line 647 + 0xf bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::execute(activemq::cmsutil::SessionCallback * action=0x0130fbf8) Line 446 C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::receive() Line 682 C++
        BrokerMonitor.exe!Receiver::ReceiveMessage() Line 95 + 0x18 bytes C++
        BrokerMonitor.exe!Receiver::run() Line 129 C++

        Call Stack 2:
        activemq-cppud.dll!activemq::core::ActiveMQConnection::~ActiveMQConnection() Line 242 C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::`vbase destructor'() + 0xf bytes C++
        activemq-cppud.dll!activemq::core::ActiveMQConnection::`vector deleting destructor'() + 0x4d bytes C++
        activemq-cppud.dll!activemq::cmsutil::ResourceLifecycleManager::destroy() Line 135 + 0x20 bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsAccessor::destroy() Line 131 C++
        activemq-cppud.dll!activemq::cmsutil::CmsDestinationAccessor::destroy() Line 58 C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::destroy() Line 211 + 0xb bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::doReceive(cms::MessageConsumer * consumer=0x01408d60) Line 632 + 0x1f bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::ReceiveExecutor::doInCms(cms::Session * session=0x01405760) Line 647 + 0xf bytes C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::execute(activemq::cmsutil::SessionCallback * action=0x0130fbf8) Line 446 C++
        activemq-cppud.dll!activemq::cmsutil::CmsTemplate::receive() Line 682 C++
        BrokerMonitor.exe!Receiver::ReceiveMessage() Line 95 + 0x18 bytes C++
        BrokerMonitor.exe!Receiver::run() Line 129 C++

        Show
        Helen Huang added a comment - - edited Hi Timothy, I have tested the new ActiveMQConnection.cpp, but got a "pure virtual function call" exception from ActiveMQConnection::CleanUp(). From the call stack, I found TransportFilter::fire was called before ActiveMQConnection::CleanUp(). msvcr80d.dll!_NMSG_WRITE(int rterrnum=25) Line 198 C msvcr80d.dll!_purecall() Line 54 + 0x7 bytes C activemq-cppud.dll!activemq::core::ActiveMQConnection::cleanup() Line 469 + 0x49 bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::onException(const decaf::lang::Exception & ex= {...}) Line 793 C++ > activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex={...} ) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex= {...}) Line 47 C++ activemq-cppud.dll!activemq::transport::inactivity::InactivityMonitor::onException(const decaf::lang::Exception & ex={...} ) Line 314 C++ activemq-cppud.dll!activemq::transport::TransportFilter::fire(const decaf::lang::Exception & ex= {...}) Line 54 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::TransportFilter::onException(const decaf::lang::Exception & ex={...} ) Line 47 C++ activemq-cppud.dll!activemq::transport::IOTransport::fire(decaf::lang::Exception & ex= {...} ) Line 72 + 0x17 bytes C++ activemq-cppud.dll!activemq::transport::IOTransport::run() Line 245 C++ activemq-cppud.dll!decaf::lang::ThreadProperties::runCallback(decaf::lang::ThreadProperties * properties=0x0131b718) Line 137 + 0x13 bytes C++ activemq-cppud.dll!`anonymous namespace'::threadWorker(void * arg=0x0131b718) Line 210 + 0x9 bytes C++ msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C msvcr80d.dll!_threadstartex(void * ptd=0x01333ba8) Line 331 C In function TransportFilter::fire(), the listener pointer was NULL at the crash. I guess some other thread might have deleted that listener/ActiveMQConnection while this thread is in ActiveMQConnection::cleanup(). void TransportFilter::fire( const decaf::lang::Exception& ex ) { if( listener != NULL ){ try { listener->onException( ex ); } catch( ... ){} } } I have put a break point in the destructor of ActiveMQConnection, and found these threads could delete it when the broker goes down: Call Stack 1: activemq-cppud.dll!activemq::core::ActiveMQConnection::~ActiveMQConnection() Line 242 C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::`vbase destructor'() + 0xf bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::`vector deleting destructor'() + 0x4d bytes C++ activemq-cppud.dll!activemq::cmsutil::ResourceLifecycleManager::destroy() Line 135 + 0x20 bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsAccessor::destroy() Line 131 C++ activemq-cppud.dll!activemq::cmsutil::CmsDestinationAccessor::destroy() Line 58 C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::destroy() Line 211 + 0xb bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::doReceive(cms::MessageConsumer * consumer=0x01408d60) Line 632 + 0x1f bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::ReceiveExecutor::doInCms(cms::Session * session=0x01405760) Line 647 + 0xf bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::execute(activemq::cmsutil::SessionCallback * action=0x0130fbf8) Line 446 C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::receive() Line 682 C++ BrokerMonitor.exe!Receiver::ReceiveMessage() Line 95 + 0x18 bytes C++ BrokerMonitor.exe!Receiver::run() Line 129 C++ Call Stack 2: activemq-cppud.dll!activemq::core::ActiveMQConnection::~ActiveMQConnection() Line 242 C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::`vbase destructor'() + 0xf bytes C++ activemq-cppud.dll!activemq::core::ActiveMQConnection::`vector deleting destructor'() + 0x4d bytes C++ activemq-cppud.dll!activemq::cmsutil::ResourceLifecycleManager::destroy() Line 135 + 0x20 bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsAccessor::destroy() Line 131 C++ activemq-cppud.dll!activemq::cmsutil::CmsDestinationAccessor::destroy() Line 58 C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::destroy() Line 211 + 0xb bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::doReceive(cms::MessageConsumer * consumer=0x01408d60) Line 632 + 0x1f bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::ReceiveExecutor::doInCms(cms::Session * session=0x01405760) Line 647 + 0xf bytes C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::execute(activemq::cmsutil::SessionCallback * action=0x0130fbf8) Line 446 C++ activemq-cppud.dll!activemq::cmsutil::CmsTemplate::receive() Line 682 C++ BrokerMonitor.exe!Receiver::ReceiveMessage() Line 95 + 0x18 bytes C++ BrokerMonitor.exe!Receiver::run() Line 129 C++
        Hide
        Timothy Bish added a comment -

        Possible fix for the issue

        Show
        Timothy Bish added a comment - Possible fix for the issue
        Hide
        Timothy Bish added a comment -

        The updated patch adds synchronization on the Resource manager to protect against multiple destroy calls in CmsTemplate

        Show
        Timothy Bish added a comment - The updated patch adds synchronization on the Resource manager to protect against multiple destroy calls in CmsTemplate
        Hide
        Timothy Bish added a comment -

        Any updates on this based on current commits, would like to start a v3.4.2 release before to long.

        Show
        Timothy Bish added a comment - Any updates on this based on current commits, would like to start a v3.4.2 release before to long.
        Hide
        Timothy Bish added a comment -

        Marking as resolved since no further feedback received, all current fixes are on trunk and in the v3.4.x branch.

        Show
        Timothy Bish added a comment - Marking as resolved since no further feedback received, all current fixes are on trunk and in the v3.4.x branch.

          People

          • Assignee:
            Timothy Bish
            Reporter:
            Helen Huang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development