Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9
    • Fix Version/s: 0.10
    • Component/s: C++ Broker
    • Labels:
      None

      Description

      This is sometimes very hard to observe. I was lucky enough to see it three times in the space of 500 iterations yesterday, but now, with same code & scripts & system, I have no repetition after 6000 iterations.

      my tree rev number is 1084895

      Here is the backtrace – all 3 have been identical:

      #0 0x0000003f8d030265 in raise (sig=<value optimized out>)
      at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
      #1 0x0000003f8d031d10 in abort () at abort.c:88
      #2 0x0000003f8d0296e6 in __assert_fail (
      assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
      file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
      function=0x2b28f9040440 "void qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)") at assert.c:78
      #3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end (this=0x2aaab60c2e50,
      cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
      #4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
      this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
      at qpid/broker/SessionState.cpp:265
      #5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn (this=0x2aaabc4157d0,
      frame=...) at qpid/broker/SessionState.cpp:355
      #6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
      this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
      #7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
      at ./qpid/framing/Handler.h:42
      #8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
      at qpid/broker/Connection.cpp:164
      #9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode (this=0x2aaabc47d240,
      buffer=<value optimized out>, size=<value optimized out>)
      at qpid/amqp_0_10/Connection.cpp:58
      #10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
      this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
      at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
      #11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff (this=0x2aaab002d9e0,
      buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
      #12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0, a0=..., a1=0x6)

      here is my broker start script:

      -------------- start script -----------------------

      #! /bin/bash

      export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs

      QPID_SRC=$TRUNK/qpid/cpp/src
      QPIDD=$

      {QPID_SRC}/.libs/qpidd

      echo $QPIDD

      rm -rf /tmp/mick
      mkdir /tmp/mick

      $QPIDD \
      --no-module-dir \
      --load-module ${QPID_SRC}

      /.libs/rdma.so \
      --data-dir /tmp/mick/data_1 \
      --auth=yes \
      --mgmt-enable=yes \
      --port 5813 \
      --log-enable info+ \
      --log-to-file /tmp/mick/qpidd_1.log \
      --log-source yes \
      --sasl-config=$

      {QPID_SRC}

      /tests/sasl_config \
      -d

      echo "started broker from $QPIDD"
      ----------- end script ------------------------

      and here is my client iterator script;

      ---------------------- start script ---------------------
      #! /bin/bash

      rm core.* ~/.qpidd/core*

      count=0
      while [ $count -lt 10000 ]
      do
      echo "==========================================="
      echo "TEST $count"
      echo "==========================================="
      sleep 1
      core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
      echo "core files: $

      {core_files}"

      if [ ${core_files}

      -gt 0 ]; then
      echo "core files found!"
      exit 1
      else
      echo "no core files found."
      fi
      ./qpid-perftest --username zig --password zig --protocol rdma --broker 20.0.40.14 --port 5813 --qt 4 --count 10
      count=$(( $count + 1 ))
      done
      ---------------------- end script -----------------------

      the sasl config directory that the broker is pointing at was created by the script cpp/src/tests/sasl_test_setup.sh

      also, you need to set "ulimit -l 131072" (at least that value) before starting the broker.

        Activity

        Hide
        Justin Ross added a comment -

        Approved for 0.10. Reviewed by Gordon Sim.

        Show
        Justin Ross added a comment - Approved for 0.10. Reviewed by Gordon Sim.
        Hide
        Gordon Sim added a comment -

        I think these fixes should be ported to 0-10 as they are regressions from the flow control change.

        Show
        Gordon Sim added a comment - I think these fixes should be ported to 0-10 as they are regressions from the flow control change.
        Hide
        Ken Giusti added a comment -

        This appears to be a very subtle race between a session en-queuing a message on a queue at the exact moment a receiver is de-queuing it.

        The bottom thread 4 backtrace below shows an inbound message in the process of being enqueued. At the moment the enqueue is completed and the queue lock is dropped, the top thread accesses the message, but the session is being closed down so the top thread attempts to "requeue" the message back. This happens before the enqueuing thread 4 can mark the async completion as done.

        I believe the fix is to remove the enqueueComplete call from requeue - this is no longer needed as this does not represent an asynchronous receive path.

        (gdb) bt
        #0 0x00000033d80329a5 in raise () from /lib64/libc.so.6
        #1 0x00000033d8034185 in abort () from /lib64/libc.so.6
        #2 0x00000033d802b935 in __assert_fail () from /lib64/libc.so.6
        #3 0x00007f9cdd1683b3 in invokeCallback (this=0x17b8f50, msg=...) at ./qpid/broker/AsyncCompletion.h:106
        #4 finishCompleter (this=0x17b8f50, msg=...) at ./qpid/broker/AsyncCompletion.h:152
        #5 enqueueComplete (this=0x17b8f50, msg=...) at ./qpid/broker/PersistableMessage.h:122
        #6 qpid::broker::Queue::requeue (this=0x17b8f50, msg=...) at qpid/broker/Queue.cpp:213
        #7 0x00007f9cdd193574 in operator() (this=<value optimized out>, requeue=<value optimized out>)
        at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_function.h:587
        #8 for_each<std::reverse_iterator<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*> >, std::const_mem\
        _fun_ref_t<void, qpid::broker::DeliveryRecord> > (this=<value optimized out>, requeue=<value optimized out>)
        at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_algo.h:4200
        #9 qpid::broker::SemanticState::recover (this=<value optimized out>, requeue=<value optimized out>) at qpid/broker/SemanticState.cpp:545
        #10 0x00007f9cdd193815 in qpid::broker::SemanticState::closed (this=0x3322e108) at qpid/broker/SemanticState.cpp:95
        #11 0x00007f9cdd1ad0bf in qpid::broker::SessionState::~SessionState (this=0x3322df30, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:98
        #12 0x00007f9cdd1ad759 in qpid::broker::SessionState::~SessionState (this=0x3322df30, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:106
        #13 0x00007f9cdd1a6d8a in ~auto_ptr (this=0x426d39f0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/backward/auto_ptr.h:168
        #14 qpid::broker::SessionHandler::handleDetach (this=0x426d39f0) at qpid/broker/SessionHandler.cpp:70
        #15 0x00007f9cdccf0eda in qpid::amqp_0_10::SessionHandler::detach (this=0x426d39f0, name="dd6ef6bc-eca9-45be-892b-0d37a2d0aca8")
        at qpid/amqp_0_10/SessionHandler.cpp:188
        #16 0x00007f9cdccc00d3 in invoke<qpid::framing::AMQP_AllOperations::SessionHandler> (this=0x7f9cda469260, body=<value optimized out>)
        at ./qpid/framing/SessionDetachBody.h:63
        #17 qpid::framing::AMQP_AllOperations::SessionHandler::Invoker::visit (this=0x7f9cda469260, body=<value optimized out>) at qpid/framing/AllInvoker.cpp:792
        #18 0x00007f9cdccef2ac in invoke<qpid::amqp_0_10::SessionHandler> (this=<value optimized out>, m=<value optimized out>) at ./qpid/framing/Invoker.h:67
        #19 qpid::amqp_0_10::SessionHandler::invoke (this=<value optimized out>, m=<value optimized out>) at qpid/amqp_0_10/SessionHandler.cpp:67
        #20 0x00007f9cdccef91f in qpid::amqp_0_10::SessionHandler::handleIn (this=0x426d39f0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:82
        #21 0x00007f9cdd110fa2 in operator() (this=0x4efac50, frame=...) at ./qpid/framing/Handler.h:42
        #22 qpid::broker::Connection::received (this=0x4efac50, frame=...) at qpid/broker/Connection.cpp:164
        #23 0x00007f9cdd0e97bd in qpid::amqp_0_10::Connection::decode (this=0x2e16bd0, buffer=<value optimized out>, size=<value optimized out>)
        at qpid/amqp_0_10/Connection.cpp:58
        #24 0x00007f9cdcd1cf4a in qpid::sys::AsynchIOHandler::readbuff (this=0x7f9ccc6cc220, buff=0x7f9c8f0a4820) at qpid/sys/AsynchIOHandler.cpp:135
        #25 0x00007f9cdcc64ee2 in operator() (this=0x7f9c35d858f0, h=...) at /usr/include/boost/function/function_template.hpp:1013
        #26 qpid::sys::posix::AsynchIO::readable (this=0x7f9c35d858f0, h=...) at qpid/sys/posix/AsynchIO.cpp:428
        #27 0x00007f9cdcd214d3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const ()
        from /root/qpid/cpp/src/.libs/libqpidcommon.so.2
        #28 0x00007f9cdcd20621 in qpid::sys::DispatchHandle::processEvent (this=0x7f9c35d858f8, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
        #29 0x00007f9cdcc70832 in process (this=0x179cab0) at ./qpid/sys/Poller.h:131
        #30 qpid::sys::Poller::run (this=0x179cab0) at qpid/sys/epoll/EpollPoller.cpp:519
        #31 0x00007f9cdcc686ca in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
        #32 0x00000033d84077e1 in start_thread () from /lib64/libpthread.so.0
        #33 0x00000033d80e153d in clone () from /lib64/libc.so.6

        (gdb) thread 4
        [Switching to thread 4 (Thread 14089)]#0 0x00000033d840e1b5 in __lll_unlock_wake () from /lib64/libpthread.so.0
        (gdb) bt
        #0 0x00000033d840e1b5 in __lll_unlock_wake () from /lib64/libpthread.so.0
        #1 0x00000033d840a707 in _L_unlock_657 () from /lib64/libpthread.so.0
        #2 0x00000033d840a66f in pthread_mutex_unlock () from /lib64/libpthread.so.0
        #3 0x00007f9cdd0eb1f6 in qpid::sys::Mutex::unlock (this=<value optimized out>) at ../include/qpid/sys/posix/Mutex.h:120
        #4 0x00007f9cdd169211 in ~ScopedLock (this=0x17b8f50, msg=<value optimized out>, isRecovery=false) at ../include/qpid/sys/Mutex.h:34
        #5 qpid::broker::Queue::push (this=0x17b8f50, msg=<value optimized out>, isRecovery=false) at qpid/broker/Queue.cpp:528
        #6 0x00007f9cdd16a3ff in qpid::broker::Queue::deliver (this=0x17b8f50, msg=...) at qpid/broker/Queue.cpp:168
        #7 0x00007f9cdd119c1f in qpid::broker::DeliverableMessage::deliverTo (this=0x7fff2d932520, queue=<value optimized out>) at qpid/broker/DeliverableMessage.cpp:33
        #8 0x00007f9cdd12dd5a in qpid::broker::Exchange::doRoute (this=0x17b3170, msg=..., b=...) at qpid/broker/Exchange.cpp:84
        #9 0x00007f9cdd11da8e in qpid::broker::DirectExchange::route (this=0x17b3110, msg=..., routingKey=<value optimized out>) at qpid/broker/DirectExchange.cpp:160
        #10 0x00007f9cdd18d321 in qpid::broker::SemanticState::route (this=0x7f9bfc1ab8b8, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:486
        #11 0x00007f9cdd18e311 in qpid::broker::SemanticState::handle (this=0x7f9bfc1ab8b8, msg=...) at qpid/broker/SemanticState.cpp:440
        #12 0x00007f9cdd1ac717 in qpid::broker::SessionState::handleContent (this=0x7f9bfc1ab6e0, frame=..., id=<value optimized out>) at qpid/broker/SessionState.cpp:262
        #13 0x00007f9cdd1acec7 in qpid::broker::SessionState::handleIn (this=0x7f9bfc1ab6e0, frame=...) at qpid/broker/SessionState.cpp:355
        #14 0x00007f9cdccef8d1 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x7f9bfc1ab500, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
        #15 0x00007f9cdd110fa2 in operator() (this=0x7f9badfb38e0, frame=...) at ./qpid/framing/Handler.h:42
        #16 qpid::broker::Connection::received (this=0x7f9badfb38e0, frame=...) at qpid/broker/Connection.cpp:164
        #17 0x00007f9cdd0e97bd in qpid::amqp_0_10::Connection::decode (this=0x7f9badfb5720, buffer=<value optimized out>, size=<value optimized out>)
        at qpid/amqp_0_10/Connection.cpp:58
        #18 0x00007f9cdcd1cf4a in qpid::sys::AsynchIOHandler::readbuff (this=0x7f9badfb3e70, buff=0x7f9badfb33d0) at qpid/sys/AsynchIOHandler.cpp:135
        #19 0x00007f9cdcc64ee2 in operator() (this=0x7f9badfb4c10, h=...) at /usr/include/boost/function/function_template.hpp:1013
        #20 qpid::sys::posix::AsynchIO::readable (this=0x7f9badfb4c10, h=...) at qpid/sys/posix/AsynchIO.cpp:428
        #21 0x00007f9cdcd214d3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const ()
        from /root/qpid/cpp/src/.libs/libqpidcommon.so.2
        #22 0x00007f9cdcd20621 in qpid::sys::DispatchHandle::processEvent (this=0x7f9badfb4c18, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
        #23 0x00007f9cdcc70832 in process (this=0x179cab0) at ./qpid/sys/Poller.h:131
        #24 qpid::sys::Poller::run (this=0x179cab0) at qpid/sys/epoll/EpollPoller.cpp:519
        #25 0x00007f9cdd0fdd82 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:385
        #26 0x000000000040d82d in QpiddBroker::execute (this=<value optimized out>, options=0x17977b0) at posix/QpiddBroker.cpp:187
        #27 0x000000000040a1e2 in main (argc=13, argv=0x7fff2d934de8) at qpidd.cpp:80
        (gdb)

        Show
        Ken Giusti added a comment - This appears to be a very subtle race between a session en-queuing a message on a queue at the exact moment a receiver is de-queuing it. The bottom thread 4 backtrace below shows an inbound message in the process of being enqueued. At the moment the enqueue is completed and the queue lock is dropped, the top thread accesses the message, but the session is being closed down so the top thread attempts to "requeue" the message back. This happens before the enqueuing thread 4 can mark the async completion as done. I believe the fix is to remove the enqueueComplete call from requeue - this is no longer needed as this does not represent an asynchronous receive path. (gdb) bt #0 0x00000033d80329a5 in raise () from /lib64/libc.so.6 #1 0x00000033d8034185 in abort () from /lib64/libc.so.6 #2 0x00000033d802b935 in __assert_fail () from /lib64/libc.so.6 #3 0x00007f9cdd1683b3 in invokeCallback (this=0x17b8f50, msg=...) at ./qpid/broker/AsyncCompletion.h:106 #4 finishCompleter (this=0x17b8f50, msg=...) at ./qpid/broker/AsyncCompletion.h:152 #5 enqueueComplete (this=0x17b8f50, msg=...) at ./qpid/broker/PersistableMessage.h:122 #6 qpid::broker::Queue::requeue (this=0x17b8f50, msg=...) at qpid/broker/Queue.cpp:213 #7 0x00007f9cdd193574 in operator() (this=<value optimized out>, requeue=<value optimized out>) at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_function.h:587 #8 for_each<std::reverse_iterator<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*> >, std::const_mem\ _fun_ref_t<void, qpid::broker::DeliveryRecord> > (this=<value optimized out>, requeue=<value optimized out>) at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/bits/stl_algo.h:4200 #9 qpid::broker::SemanticState::recover (this=<value optimized out>, requeue=<value optimized out>) at qpid/broker/SemanticState.cpp:545 #10 0x00007f9cdd193815 in qpid::broker::SemanticState::closed (this=0x3322e108) at qpid/broker/SemanticState.cpp:95 #11 0x00007f9cdd1ad0bf in qpid::broker::SessionState::~SessionState (this=0x3322df30, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:98 #12 0x00007f9cdd1ad759 in qpid::broker::SessionState::~SessionState (this=0x3322df30, __in_chrg=<value optimized out>) at qpid/broker/SessionState.cpp:106 #13 0x00007f9cdd1a6d8a in ~auto_ptr (this=0x426d39f0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.4/../../../../include/c++/4.4.4/backward/auto_ptr.h:168 #14 qpid::broker::SessionHandler::handleDetach (this=0x426d39f0) at qpid/broker/SessionHandler.cpp:70 #15 0x00007f9cdccf0eda in qpid::amqp_0_10::SessionHandler::detach (this=0x426d39f0, name="dd6ef6bc-eca9-45be-892b-0d37a2d0aca8") at qpid/amqp_0_10/SessionHandler.cpp:188 #16 0x00007f9cdccc00d3 in invoke<qpid::framing::AMQP_AllOperations::SessionHandler> (this=0x7f9cda469260, body=<value optimized out>) at ./qpid/framing/SessionDetachBody.h:63 #17 qpid::framing::AMQP_AllOperations::SessionHandler::Invoker::visit (this=0x7f9cda469260, body=<value optimized out>) at qpid/framing/AllInvoker.cpp:792 #18 0x00007f9cdccef2ac in invoke<qpid::amqp_0_10::SessionHandler> (this=<value optimized out>, m=<value optimized out>) at ./qpid/framing/Invoker.h:67 #19 qpid::amqp_0_10::SessionHandler::invoke (this=<value optimized out>, m=<value optimized out>) at qpid/amqp_0_10/SessionHandler.cpp:67 #20 0x00007f9cdccef91f in qpid::amqp_0_10::SessionHandler::handleIn (this=0x426d39f0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:82 #21 0x00007f9cdd110fa2 in operator() (this=0x4efac50, frame=...) at ./qpid/framing/Handler.h:42 #22 qpid::broker::Connection::received (this=0x4efac50, frame=...) at qpid/broker/Connection.cpp:164 #23 0x00007f9cdd0e97bd in qpid::amqp_0_10::Connection::decode (this=0x2e16bd0, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #24 0x00007f9cdcd1cf4a in qpid::sys::AsynchIOHandler::readbuff (this=0x7f9ccc6cc220, buff=0x7f9c8f0a4820) at qpid/sys/AsynchIOHandler.cpp:135 #25 0x00007f9cdcc64ee2 in operator() (this=0x7f9c35d858f0, h=...) at /usr/include/boost/function/function_template.hpp:1013 #26 qpid::sys::posix::AsynchIO::readable (this=0x7f9c35d858f0, h=...) at qpid/sys/posix/AsynchIO.cpp:428 #27 0x00007f9cdcd214d3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const () from /root/qpid/cpp/src/.libs/libqpidcommon.so.2 #28 0x00007f9cdcd20621 in qpid::sys::DispatchHandle::processEvent (this=0x7f9c35d858f8, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280 #29 0x00007f9cdcc70832 in process (this=0x179cab0) at ./qpid/sys/Poller.h:131 #30 qpid::sys::Poller::run (this=0x179cab0) at qpid/sys/epoll/EpollPoller.cpp:519 #31 0x00007f9cdcc686ca in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #32 0x00000033d84077e1 in start_thread () from /lib64/libpthread.so.0 #33 0x00000033d80e153d in clone () from /lib64/libc.so.6 (gdb) thread 4 [Switching to thread 4 (Thread 14089)] #0 0x00000033d840e1b5 in __lll_unlock_wake () from /lib64/libpthread.so.0 (gdb) bt #0 0x00000033d840e1b5 in __lll_unlock_wake () from /lib64/libpthread.so.0 #1 0x00000033d840a707 in _L_unlock_657 () from /lib64/libpthread.so.0 #2 0x00000033d840a66f in pthread_mutex_unlock () from /lib64/libpthread.so.0 #3 0x00007f9cdd0eb1f6 in qpid::sys::Mutex::unlock (this=<value optimized out>) at ../include/qpid/sys/posix/Mutex.h:120 #4 0x00007f9cdd169211 in ~ScopedLock (this=0x17b8f50, msg=<value optimized out>, isRecovery=false) at ../include/qpid/sys/Mutex.h:34 #5 qpid::broker::Queue::push (this=0x17b8f50, msg=<value optimized out>, isRecovery=false) at qpid/broker/Queue.cpp:528 #6 0x00007f9cdd16a3ff in qpid::broker::Queue::deliver (this=0x17b8f50, msg=...) at qpid/broker/Queue.cpp:168 #7 0x00007f9cdd119c1f in qpid::broker::DeliverableMessage::deliverTo (this=0x7fff2d932520, queue=<value optimized out>) at qpid/broker/DeliverableMessage.cpp:33 #8 0x00007f9cdd12dd5a in qpid::broker::Exchange::doRoute (this=0x17b3170, msg=..., b=...) at qpid/broker/Exchange.cpp:84 #9 0x00007f9cdd11da8e in qpid::broker::DirectExchange::route (this=0x17b3110, msg=..., routingKey=<value optimized out>) at qpid/broker/DirectExchange.cpp:160 #10 0x00007f9cdd18d321 in qpid::broker::SemanticState::route (this=0x7f9bfc1ab8b8, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:486 #11 0x00007f9cdd18e311 in qpid::broker::SemanticState::handle (this=0x7f9bfc1ab8b8, msg=...) at qpid/broker/SemanticState.cpp:440 #12 0x00007f9cdd1ac717 in qpid::broker::SessionState::handleContent (this=0x7f9bfc1ab6e0, frame=..., id=<value optimized out>) at qpid/broker/SessionState.cpp:262 #13 0x00007f9cdd1acec7 in qpid::broker::SessionState::handleIn (this=0x7f9bfc1ab6e0, frame=...) at qpid/broker/SessionState.cpp:355 #14 0x00007f9cdccef8d1 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x7f9bfc1ab500, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #15 0x00007f9cdd110fa2 in operator() (this=0x7f9badfb38e0, frame=...) at ./qpid/framing/Handler.h:42 #16 qpid::broker::Connection::received (this=0x7f9badfb38e0, frame=...) at qpid/broker/Connection.cpp:164 #17 0x00007f9cdd0e97bd in qpid::amqp_0_10::Connection::decode (this=0x7f9badfb5720, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #18 0x00007f9cdcd1cf4a in qpid::sys::AsynchIOHandler::readbuff (this=0x7f9badfb3e70, buff=0x7f9badfb33d0) at qpid/sys/AsynchIOHandler.cpp:135 #19 0x00007f9cdcc64ee2 in operator() (this=0x7f9badfb4c10, h=...) at /usr/include/boost/function/function_template.hpp:1013 #20 qpid::sys::posix::AsynchIO::readable (this=0x7f9badfb4c10, h=...) at qpid/sys/posix/AsynchIO.cpp:428 #21 0x00007f9cdcd214d3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const () from /root/qpid/cpp/src/.libs/libqpidcommon.so.2 #22 0x00007f9cdcd20621 in qpid::sys::DispatchHandle::processEvent (this=0x7f9badfb4c18, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280 #23 0x00007f9cdcc70832 in process (this=0x179cab0) at ./qpid/sys/Poller.h:131 #24 qpid::sys::Poller::run (this=0x179cab0) at qpid/sys/epoll/EpollPoller.cpp:519 #25 0x00007f9cdd0fdd82 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:385 #26 0x000000000040d82d in QpiddBroker::execute (this=<value optimized out>, options=0x17977b0) at posix/QpiddBroker.cpp:187 #27 0x000000000040a1e2 in main (argc=13, argv=0x7fff2d934de8) at qpidd.cpp:80 (gdb)
        Hide
        Ken Giusti added a comment -

        Most certainly caused by my implementation of https://issues.apache.org/jira/browse/QPID-2921. Investigating.

        Show
        Ken Giusti added a comment - Most certainly caused by my implementation of https://issues.apache.org/jira/browse/QPID-2921 . Investigating.

          People

          • Assignee:
            Ken Giusti
            Reporter:
            michael j. goulish
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development