Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
qpid-cpp-1.38.0
-
None
Description
Description of problem:
When below two actions happen concurrently on an auto-delete queue (without auto-del timeout), qpidd segfaults.
Two actions:
- detaching (latest) consumer of the auto-del queue
- deleting the queue in either way (i.e. via QMF or by sending proper AMQP performative)
cause that:
- segfaulting thread from the detach event has backtrace like:
#0 0x00007f9af745f40d in ScopedLock (this=0x7f9adc6c5a88, expectedVersion=1) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/Mutex.h:33
#1 qpid::broker::Queue::tryAutoDelete (this=0x7f9adc6c5a88, expectedVersion=1) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1348
#2 0x00007f9af745eed4 in qpid::broker::Queue::scheduleAutoDelete (this=0x7f9adc6c5a88, immediate=false) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1339
#3 0x00007f9af745f524 in qpid::broker::Queue::tryAutoDelete (this=0x7f9adc6c5a88, expectedVersion=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1367
#4 0x00007f9af745eed4 in qpid::broker::Queue::scheduleAutoDelete (this=0x7f9adc6c5a88, immediate=false) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1339
..
#12012 0x00007f9af745eed4 in qpid::broker::Queue::scheduleAutoDelete (this=0x7f9adc6c5a88, immediate=false) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1339
#12013 0x00007f9af74601c6 in qpid::broker::Queue::cancel (this=0x7f9adc6c5a88, c=..., connectionId="qpid.127.0.0.1:5672-127.0.0.1:56736", userId="anonymous")
at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:637
..
- other thread trying to delete the queue has bt like:
#0 0x000000343ee11016 in qpid::broker::Exchange::propagateFedOp (this=0x260ee30, routingKey="autoDel_93", tags="", op="U", origin="", extra_args=0x0)
at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Exchange.cpp:344
#1 0x000000343ee66baf in qpid::broker::DirectExchange::unbind (this=0x260edd0, queue=..., routingKey="autoDel_93", args=<value optimized out>)
at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/DirectExchange.cpp:159
#2 0x000000343eeac7c9 in qpid::broker::QueueBindings::unbind (this=<value optimized out>, exchanges=..., queue=...) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/QueueBindings.cpp:47
#3 0x000000343ee39b1d in qpid::broker::Queue::unbind (this=<value optimized out>, exchanges=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1208
#4 0x000000343ee39ca6 in qpid::broker::Queue::destroyed (this=0x7ff820308810) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1162
#5 0x000000343eeb1027 in qpid::broker::QueueRegistry::destroy (this=0x2606278, name="autoDel_93", connectionId="qpid.127.0.0.1:5672-127.0.0.1:39240", userId="anonymous")
..
The segfault happens since depth of the backtrace exceeds some limit.
Version-Release number of selected component (if applicable):
current upstream
1.36
1.38
How reproducible:
randomly, but 100% in 1 hour
Steps to Reproduce:
compile C++ program to delete a queue via QMF - attached at the botton
run below script with optionally updated parameters, such that numbers of "resource-deleted" (from qpid-receive) and "Delete failed. No such queue" (from delete_queue) errors are similar (i.e. the events to trigger the race condition happen usually at very similar time)
queues=125
slept=0.9922
for i in $(seq 1 $queues); do
while true; do
qpid-receive -a "autoDel_${i}; {create:always, node:{ x-declare:
}}" --timeout=1 &
sleep $slept
delete_queue autoDel_${i}
sleep 1
done &
sleep 0.1
done
Actual results:
within a hour, segfault with above backtraces
Expected results:
no segfault
Additional info:
delete_queue.cpp :
#include <cstdlib>
#include <iostream>
#include <sstream>
#include <qpid/messaging/Address.h>
#include <qpid/messaging/Connection.h>
#include <qpid/messaging/Message.h>
#include <qpid/messaging/Sender.h>
#include <qpid/messaging/Receiver.h>
#include <qpid/messaging/Session.h>
using namespace qpid::messaging;
using namespace qpid::types;
using std::stringstream;
using std::string;
int main(int argc, char** argv) {
const char* queue_name = argc>1 ? argv[1] : "queue_name";
const char* url = argc>2 ? argv[2] : "amqp:tcp:127.0.0.1:5672";
Connection connection(url/, connectionOptions/);
try {
connection.open();
Session session = connection.createSession();
Sender sender = session.createSender("qmf.default.direct/broker");
Address responseQueue("#reply-queue; {create:always, node:{x-declare:
}}");
Receiver receiver = session.createReceiver(responseQueue);
Message message;
Variant::Map content;
Variant::Map OID;
Variant::Map arguments;
OID["_object_name"] = "org.apache.qpid.broker:broker:amqp-broker";
arguments["type"] = "queue";
arguments["name"] = queue_name;
content["_object_id"] = OID;
content["_method_name"] = "delete";
content["_arguments"] = arguments;
encode(content, message);
message.setReplyTo(responseQueue);
message.setProperty("x-amqp-0-10.app-id", "qmf2");
message.setProperty("qmf.opcode", "_method_request");
sender.send(message, true);
Message response;
if (receiver.fetch(response,qpid::messaging::Duration(30000)) == true)
{
qpid::types::Variant::Map recv_props = response.getProperties();
if (recv_props["x-amqp-0-10.app-id"] == "qmf2")
if (recv_props["qmf.opcode"] == "_method_response")
std::cout << "Response: OK" << std::endl;
else if (recv_props["qmf.opcode"] == "_exception")
std::cerr << "Error: " << response.getContent() << std::endl;
else
std::cerr << "Invalid response received!" << std::endl;
else
std::cerr << "Invalid response not of qmf2 type received!" << std::endl;
}
else
std::cout << "Timeout: No response received within 30 seconds!" << std::endl;
connection.close();
return 0;
} catch(const std::exception& error) {
std::cout << error.what() << std::endl;
connection.close();
}
return 1;
}