Qpid
  1. Qpid
  2. QPID-4424

C++ Broker on Windows - Assertion Failed: !dispatcher - PollableQueue.h line 136

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18
    • Fix Version/s: 0.20, 0.21
    • Component/s: C++ Broker
    • Labels:
      None
    • Environment:

      Windows Server 2008 R2 64-bit system
      Visual Studio 2010 32-bit executable

      Description

      Start broker with 'qpidd --auth no --no-data-dir'.
      Sit for 15 or so seconds. No external connections from clients. No cluster.
      Assert happens.
      Stack at assert time:

      msvcr100d.dll!_NMSG_WRITE(int rterrnum=10) Line 217 C
      msvcr100d.dll!abort() Line 61 + 0x7 bytes C
      msvcr100d.dll!_wassert(const wchar_t * expr=0x5ff25504, const wchar_t * filename=0x5ff25480, unsigned int lineno=136) Line 153 C
      > qpidbrokerd.dll!qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> >::dispatch(qpid::sys::PollableCondition & cond=

      {...}) Line 136 + 0x36 bytes C++
      qpidbrokerd.dll!boost::_mfi::mf1<void,qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> >,qpid::sys::PollableCondition &>::operator()(qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> > * p=0x00c72b80, qpid::sys::PollableCondition & a1={...}

      ) Line 165 + 0x10 bytes C++
      qpidbrokerd.dll!boost::_bi::list2<boost::_bi::value<qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> > *>,boost::arg<1> >::operator()<boost::_mfi::mf1<void,qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> >,qpid::sys::PollableCondition &>,boost::_bi::list1<qpid::sys::PollableCondition &> >(boost::_bi::type<void> __formal=

      {...}, boost::_mfi::mf1<void,qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> >,qpid::sys::PollableCondition &> & f={...}

      , boost::_bi::list1<qpid::sys::PollableCondition &> & a=

      {...}, boost::_bi::type<void> __formal={...}

      ) Line 314 C++
      qpidbrokerd.dll!boost::_bi::bind_t<void,boost::_mfi::mf1<void,qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> >,qpid::sys::PollableCondition &>,boost::_bi::list2<boost::_bi::value<qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> > *>,boost::arg<1> > >::operator()<qpid::sys::PollableCondition>(qpid::sys::PollableCondition & a1=

      {...}) Line 33 C++
      qpidbrokerd.dll!boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,boost::_mfi::mf1<void,qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> >,qpid::sys::PollableCondition &>,boost::_bi::list2<boost::_bi::value<qpid::sys::PollableQueue<std::pair<boost::shared_ptr<qpid::broker::Exchange>,qpid::broker::Message> > *>,boost::arg<1> > >,void,qpid::sys::PollableCondition &>::invoke(boost::detail::function::function_buffer & function_obj_ptr={...}

      , qpid::sys::PollableCondition & a0=

      {...}) Line 154 C++
      qpidcommond.dll!boost::function1<void,qpid::sys::PollableCondition &>::operator()(qpid::sys::PollableCondition & a0={...}

      ) Line 760 + 0x1a bytes C++
      qpidcommond.dll!qpid::sys::PollableConditionPrivate::dispatch(qpid::sys::windows::AsynchIoResult * result=0x00cad6e0) Line 81 C++
      qpidcommond.dll!boost::_mfi::mf1<void,qpid::sys::PollableConditionPrivate,qpid::sys::windows::AsynchIoResult *>::operator()(qpid::sys::PollableConditionPrivate * p=0x00c72cd0, qpid::sys::windows::AsynchIoResult * a1=0x00cad6e0) Line 165 + 0x10 bytes C++
      qpidcommond.dll!boost::_bi::list2<boost::_bi::value<qpid::sys::PollableConditionPrivate *>,boost::arg<1> >::operator()<boost::_mfi::mf1<void,qpid::sys::PollableConditionPrivate,qpid::sys::windows::AsynchIoResult *>,boost::_bi::list1<qpid::sys::windows::AsynchIoResult * &> >(boost::_bi::type<void> __formal=

      {...}, boost::_mfi::mf1<void,qpid::sys::PollableConditionPrivate,qpid::sys::windows::AsynchIoResult *> & f={...}

      , boost::_bi::list1<qpid::sys::windows::AsynchIoResult * &> & a=

      {...}, boost::_bi::type<void> __formal={...}

      ) Line 314 C++
      qpidcommond.dll!boost::_bi::bind_t<void,boost::_mfi::mf1<void,qpid::sys::PollableConditionPrivate,qpid::sys::windows::AsynchIoResult *>,boost::_bi::list2<boost::_bi::value<qpid::sys::PollableConditionPrivate *>,boost::arg<1> > >::operator()<qpid::sys::windows::AsynchIoResult *>(qpid::sys::windows::AsynchIoResult * & a1=0x00cad6e0) Line 33 C++
      qpidcommond.dll!boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,boost::_mfi::mf1<void,qpid::sys::PollableConditionPrivate,qpid::sys::windows::AsynchIoResult *>,boost::_bi::list2<boost::_bi::value<qpid::sys::PollableConditionPrivate *>,boost::arg<1> > >,void,qpid::sys::windows::AsynchIoResult *>::invoke(boost::detail::function::function_buffer & function_obj_ptr=

      {...}, qpid::sys::windows::AsynchIoResult * a0=0x00cad6e0) Line 154 C++
      qpidcommond.dll!boost::function1<void,qpid::sys::windows::AsynchIoResult *>::operator()(qpid::sys::windows::AsynchIoResult * a0=0x00cad6e0) Line 760 + 0x1a bytes C++
      qpidcommond.dll!qpid::sys::windows::AsynchCallbackRequest::complete() Line 189 C++
      qpidcommond.dll!qpid::sys::windows::AsynchResult::success(unsigned int bytesTransferred=0) Line 57 + 0xf bytes C++
      qpidcommond.dll!qpid::sys::Poller::wait(qpid::sys::Duration timeout={...}

      ) Line 196 + 0x12 bytes C++
      qpidcommond.dll!qpid::sys::Poller::run() Line 118 C++
      qpidcommond.dll!qpid::sys::Dispatcher::run() Line 37 + 0x1d bytes C++
      qpidbrokerd.dll!qpid::broker::Broker::run() Line 429 + 0xb bytes C++
      qpidd.exe!qpid::broker::QpiddBroker::execute(qpid::broker::QpiddOptions * options=0x00c5b818) Line 481 + 0x26 bytes C++
      qpidd.exe!qpid::broker::run_broker(int argc=4, char * * argv=0x00c519f8, bool hidden=false) Line 106 + 0x16 bytes C++
      qpidd.exe!main(int argc=4, char * * argv=0x00c519f8) Line 507 + 0xf bytes C++
      qpidd.exe!__tmainCRTStartup() Line 555 + 0x19 bytes C
      qpidd.exe!mainCRTStartup() Line 371 C
      kernel32.dll!75d333aa()
      [Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
      ntdll.dll!770c9ef2()
      ntdll.dll!770c9ec5()

        Activity

        Hide
        Steve Huston added a comment -

        Merged to 0.20 at r1413945

        Show
        Steve Huston added a comment - Merged to 0.20 at r1413945
        Hide
        Justin Ross added a comment -
        Show
        Justin Ross added a comment - https://reviews.apache.org/r/8147/ Approved for 0.20.
        Hide
        Steve Huston added a comment -

        Fixed on trunk r1413889.

        Requesting permission to merge to 0.20 branch. The change was reviewed on review board by Andrew Stitcher and Cliff Jansen.

        Show
        Steve Huston added a comment - Fixed on trunk r1413889. Requesting permission to merge to 0.20 branch. The change was reviewed on review board by Andrew Stitcher and Cliff Jansen.
        Hide
        Steve Huston added a comment -

        It appears that PollableQueue makes an assumption that once a thread enters dispatch() and process() that no other threads will call dispatch() until the first one finishes. This assumption holds with (and may have been made because of) the behavior in the epoll poller which drives the PollableQueue off of a handle which isn't "re-armed" until the call to dispatch() completes. However, on Windows setting the condition simply queues an I/O completion event which may be handled immediately. So on Windows it is likely that multiple threads will end up in dispatch() simultaneously. Since PollableQueue::process() releases the lock while calling callback() another thread can enter dispatch() while the previous batch is being processed.

        The question is should multiple threads be allowed to dispatch events at the same time? If so, the ordering of the callbacks may not match the order in which they were enqueued. And if events remain after callback, the re-enqueueing them may change the original order. I'm open to discussion of what the desired behavior should be, but for now I'm going to propose a new patch that prevents multiple threads from entering dispatch() at the same time, making the Windows behavior the same as on Linux.

        Show
        Steve Huston added a comment - It appears that PollableQueue makes an assumption that once a thread enters dispatch() and process() that no other threads will call dispatch() until the first one finishes. This assumption holds with (and may have been made because of) the behavior in the epoll poller which drives the PollableQueue off of a handle which isn't "re-armed" until the call to dispatch() completes. However, on Windows setting the condition simply queues an I/O completion event which may be handled immediately. So on Windows it is likely that multiple threads will end up in dispatch() simultaneously. Since PollableQueue::process() releases the lock while calling callback() another thread can enter dispatch() while the previous batch is being processed. The question is should multiple threads be allowed to dispatch events at the same time? If so, the ordering of the callbacks may not match the order in which they were enqueued. And if events remain after callback, the re-enqueueing them may change the original order. I'm open to discussion of what the desired behavior should be, but for now I'm going to propose a new patch that prevents multiple threads from entering dispatch() at the same time, making the Windows behavior the same as on Linux.
        Hide
        Andrew Stitcher added a comment -

        I think this regression was caused by r1404590.

        A work around for this bug is to run the broker without management (-m0)

        Show
        Andrew Stitcher added a comment - I think this regression was caused by r1404590. A work around for this bug is to run the broker without management (-m0)

          People

          • Assignee:
            Steve Huston
            Reporter:
            Chuck Rolke
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development