Clustered broker crashes in assertion in cluster/ExpiryPolicy.cpp

From users@qpid.apache.org:

We have a Java node connecting to one of two Qpid brokers (0.5 or 0.6,
doesn't matter, we get the same) as a client of another Java node which acts
as the server. The client sends messages over a queue, awaiting a response
message from the server on a temporary queue. Both brokers use clustering
(and are part of the same cluster). At some point, after 1 to 3 exchanges
between the client and server nodes, both Qpid brokers simply crash, without
any error being logged. Broker logfile is attached. The typical output that
appears in the broker log files - the last line in the log is always the
same, so it seems that the place at which the failure occurs is consistent.

Original message with logs and config: http://apache-qpid-users.2158936.n2.nabble.com/Qpid-clustering-broker-crash-tt5511801.html#a5511801

Stack trace:

[root@ca-mtl1-tric-01 .qpidd]# gdb /opt/qpid/current/sbin/qpidd core.6709
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/qpid/qpid-0.5/sbin/qpidd...done.
[New Thread 6732]
[New Thread 6727]
[New Thread 6721]
[New Thread 6710]
[New Thread 6709]

warning: .dynamic section for "/usr/lib/libplc4.so" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib/libplds4.so" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib/openais/libcpg.so.2" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /opt/qpid/current/lib/libqpidbroker.so.0...done.
Loaded symbols for /opt/qpid/current/lib/libqpidbroker.so.0
Reading symbols from /opt/qpid/current/lib/libqpidcommon.so.0...done.
Loaded symbols for /opt/qpid/current/lib/libqpidcommon.so.0
Reading symbols from /usr/lib/libboost_program_options.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libboost_program_options.so.2
Reading symbols from /usr/lib/libboost_filesystem.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libboost_filesystem.so.2
Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /opt/qpid/current/lib/qpid/daemon/replicating_listener.so...done.
Loaded symbols for /opt/qpid/current/lib/qpid/daemon/replicating_listener.so
Reading symbols from /opt/qpid/current/lib/qpid/daemon/replication_exchange.so...done.
Loaded symbols for /opt/qpid/current/lib/qpid/daemon/replication_exchange.so
Reading symbols from /opt/qpid/current/lib/qpid/daemon/acl.so...done.
Loaded symbols for /opt/qpid/current/lib/qpid/daemon/acl.so
Reading symbols from /opt/qpid/current/lib/qpid/daemon/ssl.so...done.
Loaded symbols for /opt/qpid/current/lib/qpid/daemon/ssl.so
Reading symbols from /opt/qpid/current/lib/libsslcommon.so.0...done.
Loaded symbols for /opt/qpid/current/lib/libsslcommon.so.0
Reading symbols from /usr/lib/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libnss3.so
Reading symbols from /usr/lib/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libssl3.so
Reading symbols from /usr/lib/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libnspr4.so
Reading symbols from /usr/lib/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libnssutil3.so
Reading symbols from /usr/lib/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libplc4.so
Reading symbols from /usr/lib/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libplds4.so
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /opt/qpid/current/lib/qpid/daemon/cluster.so...done.
Loaded symbols for /opt/qpid/current/lib/qpid/daemon/cluster.so
Reading symbols from /usr/lib/openais/libcpg.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/openais/libcpg.so.2
Reading symbols from /usr/lib/libcman.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcman.so.2
Reading symbols from /opt/qpid/current/lib/libqpidclient.so.0...done.
Loaded symbols for /opt/qpid/current/lib/libqpidclient.so.0
Reading symbols from /opt/qpid/current/lib/qpid/client/rdmaconnector.so...done.
Loaded symbols for /opt/qpid/current/lib/qpid/client/rdmaconnector.so
Reading symbols from /opt/qpid/current/lib/librdmawrap.so.0...done.
Loaded symbols for /opt/qpid/current/lib/librdmawrap.so.0
Reading symbols from /usr/lib/librdmacm.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/librdmacm.so.1
Reading symbols from /usr/lib/libibverbs.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libibverbs.so.1
Reading symbols from /opt/qpid/current/lib/qpid/client/sslconnector.so...done.
Loaded symbols for /opt/qpid/current/lib/qpid/client/sslconnector.so
Reading symbols from /opt/qpid/current/lib/qpid/daemon/rdma.so...done.
Loaded symbols for /opt/qpid/current/lib/qpid/daemon/rdma.so
Core was generated by `/opt/qpid/current/sbin/qpidd -p 10004 --cluster-name=TEST_CL --no-data-dir --au'.
Program terminated with signal 6, Aborted.
#0 0x007cd402 in __kernel_vsyscall ()
(gdb) thread apply all backtrace

Thread 6 (Thread 6709):
#0 0x007cd402 in __kernel_vsyscall ()
#1 0x00993486 in epoll_wait () from /lib/libc.so.6
#2 0x00475584 in qpid::sys::Poller::wait (this=0x8b70430, timeout=...)
at qpid/sys/epoll/EpollPoller.cpp:432
#3 0x004763d3 in qpid::sys::Poller::run (this=0x8b70430)
at qpid/sys/epoll/EpollPoller.cpp:398
#4 0x004c7ef4 in qpid::sys::Dispatcher::run (this=0xbfab0ea0)
at qpid/sys/Dispatcher.cpp:37
#5 0x001d3f21 in qpid::broker::Broker::run (this=0x8b837b0)
at qpid/broker/Broker.cpp:319
#6 0x08050155 in QpiddDaemon::child (this=0xbfab116c)
at posix/QpiddBroker.cpp:128
#7 0x002140db in qpid::broker::Daemon::fork (this=0xbfab116c)
at qpid/broker/Daemon.cpp:85
#8 0x0804ea7c in QpiddBroker::execute (this=0xbfab13f5, options=0x8b81248)
at posix/QpiddBroker.cpp:158
#9 0x0804c8e7 in main (argc=14, argv=0xbfab14a4) at qpidd.cpp:77

Thread 5 (Thread 6710):
#0 0x007cd402 in __kernel_vsyscall ()
#1 0x00706ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
--Type <return> to continue, or q <return> to quit--
from /lib/libpthread.so.0
#2 0x0099f524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3 0x0029db60 in qpid::broker::Timer::run (this=0xb7fae0b0)
at ./qpid/sys/posix/Condition.h:69
#4 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0xb7fae0b0)
at qpid/sys/posix/Thread.cpp:35
#5 0x00702832 in start_thread () from /lib/libpthread.so.0
#6 0x00992e0e in clone () from /lib/libc.so.6

Thread 4 (Thread 6721):
#0 0x007cd402 in __kernel_vsyscall ()
#1 0x00706ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#2 0x0099f524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3 0x0029db60 in qpid::broker::Timer::run (this=0x8b839b4)
at ./qpid/sys/posix/Condition.h:69
#4 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0x8b839b4)
at qpid/sys/posix/Thread.cpp:35
#5 0x00702832 in start_thread () from /lib/libpthread.so.0
#6 0x00992e0e in clone () from /lib/libc.so.6

Thread 3 (Thread 6727):
#0 0x007cd402 in __kernel_vsyscall ()
--Type <return> to continue, or q <return> to quit--
#1 0x00706ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#2 0x0099f524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3 0x0029db60 in qpid::broker::Timer::run (this=0x8b83a34)
at ./qpid/sys/posix/Condition.h:69
#4 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0x8b83a34)
at qpid/sys/posix/Thread.cpp:35
#5 0x00702832 in start_thread () from /lib/libpthread.so.0
#6 0x00992e0e in clone () from /lib/libc.so.6

Thread 2 (Thread 6732):
#0 0x007cd402 in __kernel_vsyscall ()
#1 0x00993486 in epoll_wait () from /lib/libc.so.6
#2 0x00475584 in qpid::sys::Poller::wait (this=0x8b70430, timeout=...)
at qpid/sys/epoll/EpollPoller.cpp:432
#3 0x004763d3 in qpid::sys::Poller::run (this=0x8b70430)
at qpid/sys/epoll/EpollPoller.cpp:398
#4 0x004c7ef4 in qpid::sys::Dispatcher::run (this=0xbfab0ea0)
at qpid/sys/Dispatcher.cpp:37
#5 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfab0ea0)
at qpid/sys/posix/Thread.cpp:35
#6 0x00702832 in start_thread () from /lib/libpthread.so.0
#7 0x00992e0e in clone () from /lib/libc.so.6
--Type <return> to continue, or q <return> to quit--

Thread 1 (Thread 6731):
#0 0x007cd402 in __kernel_vsyscall ()
#1 0x008e9df0 in raise () from /lib/libc.so.6
#2 0x008eb701 in abort () from /lib/libc.so.6
#3 0x008e326b in __assert_fail () from /lib/libc.so.6
#4 0x00f9cc46 in qpid::cluster::ExpiryPolicy::willExpire (this=0x8b87340,
m=...) at qpid/cluster/ExpiryPolicy.cpp:47
#5 0x0024c6c4 in qpid::broker::Message::setExpiryPolicy (this=0xb4903128,
e=...) at qpid/broker/Message.cpp:338
#6 0x0024e8bd in qpid::broker::Message::setTimestamp (this=0xb4903128, e=...)
at qpid/broker/Message.cpp:331
#7 0x0027a419 in qpid::broker::SemanticState::route (this=0x8b755b0, msg=...,
strategy=...) at qpid/broker/SemanticState.cpp:358
#8 0x0027b858 in qpid::broker::SemanticState::handle (this=0x8b755b0, msg=...)
at qpid/broker/SemanticState.cpp:348
#9 0x00292532 in qpid::broker::SessionState::handleContent (this=0x8b75488,
frame=..., id=...) at qpid/broker/SessionState.cpp:243
#10 0x00292b23 in qpid::broker::SessionState::handleIn (this=0x8b75488,
frame=...) at qpid/broker/SessionState.cpp:321
#11 0x0029640b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::ha--Type <return> to continue, or q <return> to quit--
ndle (this=0x8b75588, t=...) at ./qpid/framing/Handler.h:67
#12 0x0049840a in qpid::amqp_0_10::SessionHandler::handleIn (this=0x8b74b78,
f=...) at qpid/amqp_0_10/SessionHandler.cpp:86
#13 0x0029640b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0x8b74b80, t=...) at ./qpid/framing/Handler.h:67
#14 0x00207fbc in operator() (this=0x8b73be4, frame=...)
at ./qpid/framing/Handler.h:42
#15 qpid::broker::Connection::received (this=0x8b73be4, frame=...)
at qpid/broker/Connection.cpp:106
#16 0x00f81462 in qpid::cluster::Connection::deliveredFrame (this=0x8b73b40,
f=...) at qpid/cluster/Connection.cpp:158
#17 0x00f61b1d in qpid::cluster::Cluster::deliveredFrame (this=0x8b87470,
e=...) at qpid/cluster/Cluster.cpp:271
#18 0x00f634fd in operator() (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind/mem_fn_template.hpp:149
#19 operator()<boost::_mfi::mf1<void, qpid::cluster::Cluster, qpid::cluster::EventFrame const&>, boost::_bi::list1<qpid::cluster::EventFrame const&> > (
function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287
#20 operator()<qpid::cluster::EventFrame const> (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind/bind_template.hpp:32
#21 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,--Type <return> to continue, or q <return> to quit--
boost::_mfi::mf1<void, qpid::cluster::Cluster, qpid::cluster::EventFrame const&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::Cluster*>, boost::arg<1> > >, void, qpid::cluster::EventFrame const&>::invoke (function_obj_ptr=...,
a0=...) at /usr/include/boost/function/function_template.hpp:136
#22 0x00f6a7c4 in boost::function1<void, qpid::cluster::EventFrame const&, std::allocator<void> >::operator() (this=0x8b87908, a0=...)
at /usr/include/boost/function/function_template.hpp:576
#23 0x00f6fb79 in qpid::cluster::PollableQueue<qpid::cluster::EventFrame>::handleBatch (this=0x8b87840, values=...) at qpid/cluster/PollableQueue.h:48
#24 0x00f635fd in operator() (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind/mem_fn_template.hpp:149
#25 operator()<boost::_mfi::mf1<void, qpid::cluster::PollableQueue<qpid::cluster::EventFrame>, std::deque<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> >&>, boost::_bi::list1<std::deque<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> >&> > (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind.hpp:287
#26 operator()<std::deque<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > > (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind/bind_template.hpp:32
#27 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::cluster::PollableQueue<qpid::cluster::EventFrame>, std::deque<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> >&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::PollableQueue<qpid::clu--Type <return> to continue, or q <return> to quit--
ster::EventFrame>*>, boost::arg<1> > >, void, std::deque<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> >&>::invoke (
function_obj_ptr=..., a0=...)
at /usr/include/boost/function/function_template.hpp:136
#28 0x00f6ab94 in boost::function1<void, std::deque<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> >&, std::allocator<void> >::operator()
(this=0x8b87888, a0=...)
at /usr/include/boost/function/function_template.hpp:576
#29 0x00f7327c in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::process
(this=0x8b87840) at ./qpid/sys/PollableQueue.h:149
#30 0x00f734a2 in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::dispatch
(this=0x8b87840, cond=...) at ./qpid/sys/PollableQueue.h:136
#31 0x00f6363d in operator() (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind/mem_fn_template.hpp:149
#32 operator()<boost::_mfi::mf1<void, qpid::sys::PollableQueue<qpid::cluster::EventFrame>, qpid::sys::PollableCondition&>, boost::_bi::list1<qpid::sys::PollableCondition&> > (function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287
#33 operator()<qpid::sys::PollableCondition> (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind/bind_template.hpp:32
#34 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::PollableQueue<qpid::cluster::EventFrame>, qpid::sys::PollableCondition&>, boost::_bi::list2<boost::_bi::value<qpid::sys::PollableQueue<qpid::cluster::EventFrame>*>, boost::arg<1> > >, void, qpid::sys::Poll--Type <return> to continue, or q <return> to quit--
ableCondition&>::invoke (function_obj_ptr=..., a0=...)
at /usr/include/boost/function/function_template.hpp:136
#35 0x00473d67 in boost::function1<void, qpid::sys::PollableCondition&, std::allocator<boost::function_base> >::operator() (this=0x8b88630, a0=...)
at /usr/include/boost/function/function_template.hpp:576
#36 0x00472c37 in qpid::sys::PollableConditionPrivate::dispatch (
this=0x8b88628) at qpid/sys/posix/PollableCondition.cpp:89
#37 0x0047308d in operator() (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind/mem_fn_template.hpp:149
#38 operator()<boost::_mfi::mf1<void, qpid::sys::PollableConditionPrivate, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > (
function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287
#39 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...)
at /usr/include/boost/bind/bind_template.hpp:32
#40 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::PollableConditionPrivate, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::PollableConditionPrivate*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (
function_obj_ptr=..., a0=...)
at /usr/include/boost/function/function_template.hpp:136
#41 0x004cb547 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x8b71010, a0=...)
at /usr/include/boost/function/function_template.hpp:576
--Type <return> to continue, or q <return> to quit--
#42 0x004c9a48 in qpid::sys::DispatchHandle::processEvent (this=0x8b71008,
type=READABLE) at qpid/sys/DispatchHandle.cpp:428
#43 0x004763b0 in process (this=0x8b70430) at ./qpid/sys/Poller.h:122
#44 qpid::sys::Poller::run (this=0x8b70430)
at qpid/sys/epoll/EpollPoller.cpp:402
#45 0x004c7ef4 in qpid::sys::Dispatcher::run (this=0xbfab0ea0)
at qpid/sys/Dispatcher.cpp:37
#46 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfab0ea0)
at qpid/sys/posix/Thread.cpp:35
#47 0x00702832 in start_thread () from /lib/libpthread.so.0
#48 0x00992e0e in clone () from /lib/libc.so.6