Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.24, 0.26
-
None
-
None
-
Fedora 19
Description
Sometimes, when running qpidd it crashes on startup with the following assertion appearing in /var/log/messages
Mar 24 11:08:03 localhost qpidd[441]: qpidd: /usr/include/boost/smart_ptr/shared_ptr.hpp:653: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = qmf::org::apache::qpid::broker::Broker; typename boost::detail::sp_member_access<T>::type = qmf::org::apache::qpid::broker::Broker*]: Assertion `px != 0' failed.
Trying to increase log levels to trace+ gives us this:
2014-03-24 11:07:37 [Security] info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2014-03-24 11:07:43 [Model] trace Mgmt create memory. id:amqp-broker
2014-03-24 11:07:43 [Management] debug Management object added: amqp-broker
2014-03-24 11:07:43 [Broker] info Management enabled
2014-03-24 11:07:43 [Management] info ManagementAgent restored broker ID: e349c55c-69ef-47ad-bd53-a4c898f43e0a
2014-03-24 11:07:49 [Management] debug ManagementAgent boot sequence: 4
2014-03-24 11:07:49 [Management] debug ManagementAgent added package org.apache.qpid.broker
2014-03-24 11:07:49 [Management] debug SEND PackageInd package=org.apache.qpid.broker to=schema.package
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:system
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:memory
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:broker
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:agent
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:vhost
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:queue
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:exchange
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:binding
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:subscription
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:connection
2014-03-24 11:07:49 [Management] debug ManagementAgent added class org.apache.qpid.broker:incoming
2014-03-24 11:07:50 [Management] debug ManagementAgent added class org.apache.qpid.broker:outgoing
2014-03-24 11:07:57 [Management] debug Management agent periodic processing: management snapshot: 1 packages, 0 objects (0 deleted), 1 new objects (0 deleted), 0 pending deletes
2014-03-24 11:07:57 [Management] trace Management agent periodic processing: new objects
amqp-broker
We managed to find out what the problem was after we got a core dump of qpidd. The stack trace was:
#0 0x00007fc41f1fe9e9 in raise () from /lib64/libc.so.6
#1 0x00007fc41f2000f8 in abort () from /lib64/libc.so.6
#2 0x00007fc41f1f7956 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007fc41f1f7a02 in __assert_fail () from /lib64/libc.so.6
#4 0x00007fc4213dd0e5 in boost::shared_ptr<qmf::org::apache::qpid::broker::Broker>::operator->() const [clone .isra.49] [clone .part.50] () from /lib64/libqpidbroker.so.2
#5 0x00007fc4215edd8d in qpid::management::ManagementAgent::periodicProcessing() () from /lib64/libqpidbroker.so.2
#6 0x00007fc4215d27d0 in qpid::management::(anonymous namespace)::Periodic::fire() () from /lib64/libqpidbroker.so.2
#7 0x00007fc42102d70b in qpid::sys::Timer::fire(boost::intrusive_ptr<qpid::sys::TimerTask>) () from /lib64/libqpidcommon.so.2
#8 0x00007fc42102ed26 in qpid::sys::Timer::run() () from /lib64/libqpidcommon.so.2
#9 0x00007fc420fc20da in qpid::sys::(anonymous namespace)::runRunnable(void*) () from /lib64/libqpidcommon.so.2
#10 0x00007fc41f591c53 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fc41f2bedbd in clone () from /lib64/libc.so.6
What happens is that in Broker's constructor it calls managementAgent->configure() prior to setting mgmtObject with a proper shared_ptr. ManagementAgent::configure() starts a periodic task which tries to access mgmObject using Broker::GetManagementObject(). So it's a timing issue: if the periodic task happened to occur before setting Broker::mgmtObject, the code will crash on a null shared_ptr.
This also happens on 0.26.