Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-8319

QMF requests rerouted to QMF exchange may crash with invalid connection

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • qpid-cpp-1.39.0
    • None
    • C++ Broker
    • None

    Description

      Reported by Pavel in https://bugzilla.redhat.com/show_bug.cgi?id=1713560

      Description of problem:

      User story: when running concurrently 2 times a program that:
      1) Creates a queue on the broker "HelloQueue"
      2) Creates a second queue called "HelloQueue.AutoDelete" with auto-delete set and alternate exchange set to "qmf.default.direct" and hold open the Receiver that is subscribed to it.
      3) Puts a QMF message into the "HelloQueue.AutoDelete" queue that will delete the "HelloQueue" queue when it is processed.
      4) Waits 10 seconds.
      5) Closes the receiver, triggering the auto-delete of "HelloQueue.AutoDelete".

      Then the QMF message will be sent to "qmf.default.direct" because of the alternate exchange, resulting in the deletion of "HelloQueue" regardless of whether or not there are other subscribers connected to it. And with some high probability, the 2nd QMF request from just dropped connection will attempt to be processed, but causes segfault.

      Version-Release number of selected component (if applicable):
      qpid-cpp 1.36.0-15 (or -21 or -21+hf2), I expect any

      How reproducible:
      75% in my case

      Steps to Reproduce:
      1. Compile attached program.
      2. qpidd &
      3. ./QmfBrokerCrashRepro localhost:5672 & ./QmfBrokerCrashRepro localhost:5672 &

      Actual results:
      client program aborts every time (unhandled exception, no deal), but very often qpidd segfaults as well, with backtrace:

      (gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007f9b5cdca752 in qpid::management::(anonymous namespace)::ScopedManagementContext::getUserId (this=<value optimized out>)
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/management/ManagementAgent.cpp:105
#2  0x00007f9b5cde8055 in qpid::management::ManagementAgent::dispatchAgentCommand (this=0x1680930, msg=..., viaLocal=true)
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/management/ManagementAgent.cpp:2347
#3  0x00007f9b5cde8958 in qpid::management::ManagementAgent::dispatchCommand (this=0x1680930, deliverable=<value optimized out>, routingKey="broker", topic=false, qmfVersion=2)
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/management/ManagementAgent.cpp:1255
#4  0x00007f9b5cdfb219 in qpid::broker::ManagementDirectExchange::route (this=0x168b6f0, msg=...) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/management/ManagementDirectExchange.cpp:48
#5  0x00007f9b5cccfa2a in qpid::broker::Exchange::routeWithAlternate (this=0x168b768, msg=...) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Exchange.cpp:410
#6  0x00007f9b5ccfddb5 in qpid::broker::Queue::reroute (e=<value optimized out>, m=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1761
#7  0x00007f9b5ccfe006 in qpid::broker::Queue::abandoned (this=0x16ba740, message=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1156
#8  0x00007f9b5ccf16cd in operator() (this=0x16ba740, maxCount=0, p=..., f=..., type=<value optimized out>, triggerAutoDelete=false, maxTests=0)
    at /usr/include/boost/function/function_template.hpp:1013
#9  qpid::broker::Queue::remove (this=0x16ba740, maxCount=0, p=..., f=..., type=<value optimized out>, triggerAutoDelete=false, maxTests=0)
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:795
#10 0x00007f9b5ccf49d5 in qpid::broker::Queue::destroyed (this=0x16ba740) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1167
#11 0x00007f9b5cd73b09 in qpid::broker::QueueRegistry::destroyIfUntouched (this=0x167f2f8, targetQ=<value optimized out>, version=<value optimized out>, connectionId="", userId="")
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/QueueRegistry.cpp:156
#12 0x00007f9b5ccee336 in qpid::broker::Queue::tryAutoDelete (this=0x16ba740, expectedVersion=1) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1358
#13 0x00007f9b5ccee834 in qpid::broker::Queue::scheduleAutoDelete (this=0x16ba740, immediate=false) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:1342
#14 0x00007f9b5ccef626 in qpid::broker::Queue::cancel (this=0x16ba740, c=..., connectionId="qpid.[::1]:5672-[::1]:54658", userId="anonymous@QPID")
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/Queue.cpp:638
#15 0x00007f9b5cd90eca in qpid::broker::SemanticState::cancel (this=0x7f9b4c00a078, c=...) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/SemanticState.cpp:475
#16 0x00007f9b5cd98775 in qpid::broker::SemanticState::closed (this=0x7f9b4c00a078) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/SemanticState.cpp:111
#17 0x00007f9b5cdb0301 in qpid::broker::SessionState::~SessionState (this=0x7f9b4c009eb0, __in_chrg=<value optimized out>)
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/SessionState.cpp:107
#18 0x00007f9b5cdb08a9 in qpid::broker::SessionState::~SessionState (this=0x7f9b4c009eb0, __in_chrg=<value optimized out>)
    at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/SessionState.cpp:110
#19 0x00007f9b5cdb5c44 in ~auto_ptr (this=0x7f9b4c009d00) at /usr/include/c++/4.4.7/backward/auto_ptr.h:168
#20 qpid::broker::SessionHandler::handleDetach (this=0x7f9b4c009d00) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/SessionHandler.cpp:110
#21 0x00007f9b5cd1b564 in qpid::broker::amqp_0_10::Connection::closed (this=0x7f9b4c003e30) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/broker/amqp_0_10/Connection.cpp:378
#22 0x00007f9b5c7f374d in qpid::sys::AsynchIOHandler::disconnect (this=0x168f270) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/AsynchIOHandler.cpp:201
#23 0x00007f9b5c7f3ca9 in qpid::sys::AsynchIOHandler::eof (this=0x168f270, a=<value optimized out>) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/AsynchIOHandler.cpp:184
#24 0x00007f9b5c770e3a in operator() (this=0x168fc90, h=...) at /usr/include/boost/function/function_template.hpp:1013
#25 qpid::sys::posix::AsynchIO::readable (this=0x168fc90, h=...) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/posix/AsynchIO.cpp:486
#26 0x00007f9b5c7f79e3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#27 0x00007f9b5c7f6676 in qpid::sys::DispatchHandle::processEvent (this=0x168fc98, type=qpid::sys::Poller::READABLE) at /usr/src/debug/qpid-cpp-1.36.0/src/qpid/sys/DispatchHandle.cpp:280
..

      Here, the context (of type qpid::broker::amqp_0_10::Connection) points to the 2nd client connection that was dropped. Qpid trace logs show the connection was already closed and its management object deleted - but a reference still kept due to this QMF method..?

      Expected results:
      no segfault

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #21

          Activity

            People

              gsim Gordon Sim
              chug Charles E. Rolke
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: