Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-8209

qpidd segfault with huge backtrace when deleting autoDel queue just being auto-deleted

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: qpid-cpp-1.38.0
    • Fix Version/s: qpid-cpp-1.37.0
    • Component/s: C++ Broker
    • Labels:
      None

      Description

      Description of problem:
      When below two actions happen concurrently on an auto-delete queue (without auto-del timeout), qpidd segfaults.

      Two actions:

      • detaching (latest) consumer of the auto-del queue
      • deleting the queue in either way (i.e. via QMF or by sending proper AMQP performative)

      cause that:

      • segfaulting thread from the detach event has backtrace like:
        #0 0x00007f9af745f40d in ScopedLock (this=0x7f9adc6c5a88, expectedVersion=1) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/Mutex.h:33
        #1 qpid::broker::Queue::tryAutoDelete (this=0x7f9adc6c5a88, expectedVersion=1) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1348
        #2 0x00007f9af745eed4 in qpid::broker::Queue::scheduleAutoDelete (this=0x7f9adc6c5a88, immediate=false) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1339
        #3 0x00007f9af745f524 in qpid::broker::Queue::tryAutoDelete (this=0x7f9adc6c5a88, expectedVersion=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1367
        #4 0x00007f9af745eed4 in qpid::broker::Queue::scheduleAutoDelete (this=0x7f9adc6c5a88, immediate=false) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1339
        ..
        #12012 0x00007f9af745eed4 in qpid::broker::Queue::scheduleAutoDelete (this=0x7f9adc6c5a88, immediate=false) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1339
        #12013 0x00007f9af74601c6 in qpid::broker::Queue::cancel (this=0x7f9adc6c5a88, c=..., connectionId="qpid.127.0.0.1:5672-127.0.0.1:56736", userId="anonymous")
        at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:637
        ..
      • other thread trying to delete the queue has bt like:

      #0 0x000000343ee11016 in qpid::broker::Exchange::propagateFedOp (this=0x260ee30, routingKey="autoDel_93", tags="", op="U", origin="", extra_args=0x0)
      at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Exchange.cpp:344
      #1 0x000000343ee66baf in qpid::broker::DirectExchange::unbind (this=0x260edd0, queue=..., routingKey="autoDel_93", args=<value optimized out>)
      at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/DirectExchange.cpp:159
      #2 0x000000343eeac7c9 in qpid::broker::QueueBindings::unbind (this=<value optimized out>, exchanges=..., queue=...) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/QueueBindings.cpp:47
      #3 0x000000343ee39b1d in qpid::broker::Queue::unbind (this=<value optimized out>, exchanges=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1208
      #4 0x000000343ee39ca6 in qpid::broker::Queue::destroyed (this=0x7ff820308810) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1162
      #5 0x000000343eeb1027 in qpid::broker::QueueRegistry::destroy (this=0x2606278, name="autoDel_93", connectionId="qpid.127.0.0.1:5672-127.0.0.1:39240", userId="anonymous")
      ..

      The segfault happens since depth of the backtrace exceeds some limit.

      Version-Release number of selected component (if applicable):
      current upstream
      1.36
      1.38
      How reproducible:
      randomly, but 100% in 1 hour

      Steps to Reproduce:
      compile C++ program to delete a queue via QMF - attached at the botton
      run below script with optionally updated parameters, such that numbers of "resource-deleted" (from qpid-receive) and "Delete failed. No such queue" (from delete_queue) errors are similar (i.e. the events to trigger the race condition happen usually at very similar time)

      queues=125
      slept=0.9922
      for i in $(seq 1 $queues); do
      while true; do
      qpid-receive -a "autoDel_${i}; {create:always, node:{ x-declare:

      {auto-delete:True}

      }}" --timeout=1 &
      sleep $slept
      delete_queue autoDel_${i}
      sleep 1
      done &
      sleep 0.1
      done

      Actual results:
      within a hour, segfault with above backtraces

      Expected results:
      no segfault

      Additional info:
      delete_queue.cpp :

      #include <cstdlib>
      #include <iostream>

      #include <sstream>

      #include <qpid/messaging/Address.h>
      #include <qpid/messaging/Connection.h>
      #include <qpid/messaging/Message.h>
      #include <qpid/messaging/Sender.h>
      #include <qpid/messaging/Receiver.h>
      #include <qpid/messaging/Session.h>

      using namespace qpid::messaging;
      using namespace qpid::types;

      using std::stringstream;
      using std::string;

      int main(int argc, char** argv) {
      const char* queue_name = argc>1 ? argv[1] : "queue_name";
      const char* url = argc>2 ? argv[2] : "amqp:tcp:127.0.0.1:5672";

      Connection connection(url/, connectionOptions/);
      try {
      connection.open();
      Session session = connection.createSession();
      Sender sender = session.createSender("qmf.default.direct/broker");
      Address responseQueue("#reply-queue; {create:always, node:{x-declare:

      {auto-delete:true}

      }}");
      Receiver receiver = session.createReceiver(responseQueue);

      Message message;
      Variant::Map content;
      Variant::Map OID;
      Variant::Map arguments;
      OID["_object_name"] = "org.apache.qpid.broker:broker:amqp-broker";
      arguments["type"] = "queue";
      arguments["name"] = queue_name;

      content["_object_id"] = OID;
      content["_method_name"] = "delete";
      content["_arguments"] = arguments;

      encode(content, message);
      message.setReplyTo(responseQueue);
      message.setProperty("x-amqp-0-10.app-id", "qmf2");
      message.setProperty("qmf.opcode", "_method_request");

      sender.send(message, true);

      Message response;
      if (receiver.fetch(response,qpid::messaging::Duration(30000)) == true)
      {
      qpid::types::Variant::Map recv_props = response.getProperties();
      if (recv_props["x-amqp-0-10.app-id"] == "qmf2")
      if (recv_props["qmf.opcode"] == "_method_response")
      std::cout << "Response: OK" << std::endl;
      else if (recv_props["qmf.opcode"] == "_exception")
      std::cerr << "Error: " << response.getContent() << std::endl;
      else
      std::cerr << "Invalid response received!" << std::endl;
      else
      std::cerr << "Invalid response not of qmf2 type received!" << std::endl;
      }
      else
      std::cout << "Timeout: No response received within 30 seconds!" << std::endl;

      connection.close();
      return 0;
      } catch(const std::exception& error) {
      std::cout << error.what() << std::endl;
      connection.close();
      }
      return 1;
      }

        Attachments

          Activity

            People

            • Assignee:
              cliffjansen Cliff Jansen
              Reporter:
              pmoravec Pavel Moravec
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: