Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Cannot Reproduce
-
2.1.3
-
None
-
None
-
Windows XP/Server 2003
-
Regression
Description
This problems was seen on:
Versions:
ActiveMQ-cpp-2.1.3 (WindowsServer2003/XP)
ActiveMQ Broker 5.1 (WindowsServer2003)
This looks similar to issue AMQCPP-122, which was fixed in 2.1, but I don't see how IOTransport::run() and error handling have been properly synchronized.
Steps to reproduce:
- Continuously try to reconnect to an activemq broker that has run out of memory.
- This may eventually produce the crash (could take several hours to produce depending on frequency of reconnect attempts).
- Running activemq-cpp through purify can help reproduce this problem more easily.
- A "R6025 pure virtual function call" error message may be printed out to the console when this error happens.
Backtraces:
Thread 1:
activemq::transport::TransportFilter::fire() + 0x48 bytes activemq::transport::TransportFilter::fire() + 0x48 bytes activemq::transport::IOTransport::fire() + 0x48 bytes activemq::transport::IOTransport::run() + 0x7f bytes activemq::concurrent::Thread::runCallback() + 0x45 bytes msvcr80.dll!781329bb()
The crash happens on this line:
exceptionListener->onTransportException( this, ex );
Thread 2:
activemq::concurrent::Thread::join() Line 108 C++ activemq::transport::IOTransport::close() Line 143 C++ activemq::transport::TransportFilter::close() Line 213 C++ activemq::transport::filters::TcpTransport::close() Line 143 + 0xb bytes C++ activemq::transport::filters::ResponseCorrelator::close() Line 241 C++ activemq::transport::filters::ResponseCorrelator::~ResponseCorrelator() Line 64 C++ activemq::transport::filters::ResponseCorrelator::`scalar deleting destructor'() + 0xf bytes C++ activemq::core::ActiveMQConnectionFactory::createConnection(const std::basic_string<char,std::char_traits<char>,std::allocator<char> activemq::core::ActiveMQConnectionFactory::createConnection() Line 66 + 0x3a bytes C++
During ActiveMQConnectionFactory::createConnection() an exception is thrown and the transport object is deleted. Unfortunately,
while being deleted this object is still being used by Thread#1 (IOTransport::run).
I greatly reduced the likelihood of this problem by calling setTransportExceptionListener(NULL) in TransportFilter's destructor.
After doing that, another crash will start to appear (under the same test conditions) with the following backtrace:
Thread 1:
activemq::connector::openwire::OpenWireCommandReader::readCommand() Line 71 + 0x1e bytes C++ activemq::transport::IOTransport::run() Line 166 + 0x19 bytes C++ activemq::concurrent::Thread::runCallback(void * param=0x02a750b0) Line 152 + 0x13 bytes C++ msvcr80d.dll!_callthreadstartex() Line 348 + 0xf bytes C msvcr80d.dll!_threadstartex(void * ptd=0x02a6b8c0) Line 331 C kernel32.dll!7c80b683() ntdll.dll!7c91b686()
The crash happens on this line:
return openWireFormat->unmarshal( dataInputStream );
Thread 2:
activemq::concurrent::Thread::join() Line 108 C++ activemq::transport::IOTransport::close() Line 143 C++ activemq::transport::TransportFilter::close() Line 213 C++ activemq::transport::filters::TcpTransport::close() Line 143 + 0xb bytes C++ activemq::transport::filters::ResponseCorrelator::close() Line 241 C++ activemq::transport::filters::ResponseCorrelator::~ResponseCorrelator() Line 64 C++ activemq::transport::filters::ResponseCorrelator::`scalar deleting destructor'() + 0xf bytes C++ activemq::core::ActiveMQConnectionFactory::createConnection(const std::basic_string<char,std::char_traits<char>,std::allocator<char> activemq::core::ActiveMQConnectionFactory::createConnection() Line 66 + 0x3a bytes C++
This second problem is similar to the first and seems to be caused when the OpenWireConnector is deleted before IOTransport::close() is called. Since IOTransport::run() tries to use the OpenWireConnector (via OpenWireCommandReader::readCommand()), a crash can occur.