Qpid
  1. Qpid
  2. QPID-3893

C++ broker appears to segfault during MultipleTransactedBatchProducerTest

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.15, 0.16, 0.17
    • Fix Version/s: 0.16, 0.17
    • Component/s: C++ Broker
    • Labels:
      None

      Description

      The C++ broker appears to segfault during MultipleTransactedBatchProducerTest. Several instances can be found in the history from our CI jobs:

      https://builds.apache.org/job/Qpid-Java-Cpp-Test/496/
      https://builds.apache.org/job/Qpid-Java-Cpp-Test/494/
      https://builds.apache.org/job/Qpid-Java-Cpp-Test/493/
      https://builds.apache.org/job/Qpid-Java-Cpp-Test/491/
      https://builds.apache.org/job/Qpid-Java-Cpp-Test/486/
      https://builds.apache.org/job/Qpid-Java-Cpp-Test/484/
      https://builds.apache.org/job/Qpid-Java-Cpp-Test/482/

      During course of the latest test failure in #496 (logs attached), the broker logged receiving yet another TxCommit command:

      BROKER: 2012-03-11 16:55:15 trace RECV [127.0.0.1:15672-127.0.0.1:34582]: Frame[BEbe; channel=0; {TxCommitBody: }] )
      

      After this nothing else is ever logged from the broker process, but exceptions start flying from the client indicating the connections were reset:

      IoReceiver - localhost/127.0.0.1:15672 2012-03-11 16:55:15,313 ERROR [apache.qpid.client.AMQConnectionDelegate_0_10] previous exception
      org.apache.qpid.transport.ConnectionException: Connection reset
      	at org.apache.qpid.transport.Connection.exception(Connection.java:512)
      	at org.apache.qpid.transport.network.Assembler.exception(Assembler.java:107)
      	at org.apache.qpid.transport.network.InputHandler.exception(InputHandler.java:199)
      	at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:169)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.net.SocketException: Connection reset
      	at java.net.SocketInputStream.read(SocketInputStream.java:168)
      	at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:147)
      	... 1 more
      
      IoSender - localhost/127.0.0.1:15672 2012-03-11 16:55:15,313 ERROR [transport.network.io.IoSender] error in write thread
      java.net.SocketException: Connection reset
      	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
      	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      	at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:313)
      	at java.lang.Thread.run(Thread.java:662)
      

      When the test tearDown occurs, it is logged that the broker process exited with an abnormal looking exit code, which a quick google suggests is the result of a segfault:

      main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.QpidBrokerTestCase] stopping broker on port : 15672
      main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.SpawnedBrokerHolder] Destroying broker process
      main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.SpawnedBrokerHolder] broker exited: 139
      

      The test fails after 75sec, which is the exact time the test itself allows for the process undertaken to complete (artificially high to allow for extreme slowdowns on the CI nodes which happened occasionally in the past and also the C++ broker being slower at the test due to use of client side selectors; passing results show this usually takes 1-3sec on the Java broker and <10sec on the [transient] C++ broker), however the test harness tries to clean up the connections use in the test during tearDown and actually ends up reporting issues cleaning those up rather than the fact that the CountDownLatch used in the test wasnt decremented enough times.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        11d 1h 24m 1 Andrew Stitcher 22/Mar/12 19:12
        Resolved Resolved Closed Closed
        493d 23h 40m 1 Justin Ross 29/Jul/13 19:52
        Justin Ross made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Andrew Stitcher made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.16 [ 12319870 ]
        Resolution Fixed [ 1 ]
        Hide
        Andrew Stitcher added a comment -

        Changes checked into 0.16 release branch

        Show
        Andrew Stitcher added a comment - Changes checked into 0.16 release branch
        Hide
        Justin Ross added a comment -

        Reviewed by Chuck. Approved for 0.16.

        Show
        Justin Ross added a comment - Reviewed by Chuck. Approved for 0.16.
        Hide
        Chuck Rolke added a comment -

        Ship it! Part 1 locking makes sense and part 2 boost namespace compiles on windows.

        Show
        Chuck Rolke added a comment - Ship it! Part 1 locking makes sense and part 2 boost namespace compiles on windows.
        Andrew Stitcher made changes -
        Hide
        Andrew Stitcher added a comment -

        Necessary extra code changes for the fix to compile under windows

        Show
        Andrew Stitcher added a comment - Necessary extra code changes for the fix to compile under windows
        Andrew Stitcher made changes -
        Hide
        Andrew Stitcher added a comment -

        Fix for this issue

        Show
        Andrew Stitcher added a comment - Fix for this issue
        Hide
        Andrew Stitcher added a comment -

        Checked in fix to trunk

        Show
        Andrew Stitcher added a comment - Checked in fix to trunk
        Andrew Stitcher made changes -
        Fix Version/s 0.17 [ 12320179 ]
        Andrew Stitcher made changes -
        Assignee Andrew Stitcher [ astitcher ]
        Hide
        Gordon Sim added a comment -
        Program terminated with signal 11, Segmentation fault.
        #0  qpid::framing::FieldValue::print (this=0x2aaab003acc0, out=...) at ../../src/qpid/framing/FieldValue.cpp:223
        223	    if (data->convertsToString()) out << data->getString();
        (gdb) bt
        #0  qpid::framing::FieldValue::print (this=0x2aaab003acc0, out=...) at ../../src/qpid/framing/FieldValue.cpp:223
        #1  0x00002b0f042fec09 in operator<< (out=<value optimized out>, i=<value optimized out>) at ../../include/qpid/framing/FieldValue.h:136
        #2  qpid::framing::(anonymous namespace)::operator<< (out=<value optimized out>, i=<value optimized out>) at ../../src/qpid/framing/FieldTable.cpp:78
        #3  0x00002b0f04300d3b in qpid::framing::operator<< (out=..., t=...) at ../../src/qpid/framing/FieldTable.cpp:86
        #4  0x00002b0f04277bc0 in qpid::framing::MessageProperties::print (this=0x2aaab004dd78, out=...) at qpid/framing/MessageProperties.cpp:221
        #5  0x00002b0f04277daf in qpid::framing::operator<< (out=..., s=...) at qpid/framing/MessageProperties.cpp:229
        #6  0x00002b0f042f218e in print (this=0x2aaab004dd10, out=...) at ../../src/qpid/framing/AMQHeaderBody.h:64
        #7  qpid::framing::AMQHeaderBody::print (this=0x2aaab004dd10, out=...) at ../../src/qpid/framing/AMQHeaderBody.cpp:57
        #8  0x00002b0f042f0360 in qpid::framing::operator<< (out=..., body=...) at ../../src/qpid/framing/AMQBody.cpp:34
        #9  0x00002b0f042e2e5d in operator<< <qpid::framing::AMQBody> (this=0x2aaab00582e0, f=...) at ../../include/qpid/Msg.h:63
        #10 qpid::SessionState::senderRecord (this=0x2aaab00582e0, f=...) at ../../src/qpid/SessionState.cpp:119
        #11 0x00002b0f042ebc85 in qpid::amqp_0_10::SessionHandler::handleOut (this=0x2aaab0105a10, f=...) at ../../src/qpid/amqp_0_10/SessionHandler.cpp:142
        #12 0x00002b0f03d9b720 in map_if<qpid::framing::Relay, qpid::framing::TypeFilter<2u> > (this=0x2aaab81325e0, out=...) at ../../src/qpid/framing/frame_functors.h:90
        #13 qpid::broker::Message::sendHeader (this=0x2aaab81325e0, out=...) at ../../src/qpid/broker/Message.cpp:291
        #14 0x00002b0f03d5a8a3 in qpid::broker::DeliveryRecord::deliver (this=0x487043f0, h=..., deliveryId=<value optimized out>, framesize=65535) at ../../src/qpid/broker/DeliveryRecord.cpp:89
        #15 0x00002b0f03e1c7d4 in qpid::broker::SessionState::deliver (this=0x2aaab00582e0, msg=..., sync=false) at ../../src/qpid/broker/SessionState.cpp:302
        #16 0x00002b0f03df9e70 in qpid::broker::SemanticState::ConsumerImpl::deliver (this=0x2aaab8136500, msg=...) at ../../src/qpid/broker/SemanticState.cpp:348
        #17 0x00002b0f03dbf2e6 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at ../../src/qpid/broker/Queue.cpp:475
        #18 0x00002b0f03df7dcc in qpid::broker::SemanticState::ConsumerImpl::doDispatch (this=<value optimized out>) at ../../src/qpid/broker/SemanticState.cpp:664
        #19 0x00002b0f03df71f9 in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x2aaab8136500) at ../../src/qpid/broker/SemanticState.cpp:731
        #20 0x00002b0f0431cdf3 in qpid::sys::AggregateOutput::doOutput (this=<value optimized out>) at ../../src/qpid/sys/AggregateOutput.cpp:59
        #21 0x00002b0f03d4b509 in qpid::broker::Connection::doOutput (this=0x2aaab005a3e0) at ../../src/qpid/broker/Connection.cpp:360
        #22 0x00002b0f03d19ca0 in qpid::amqp_0_10::Connection::canEncode (this=0x2aaab0104240) at ../../src/qpid/amqp_0_10/Connection.cpp:67
        #23 0x00002b0f03df61ba in qpid::broker::SecureConnection::canEncode (this=<value optimized out>) at ../../src/qpid/broker/SecureConnection.cpp:58
        #24 0x00002b0f04320480 in qpid::sys::AsynchIOHandler::idle (this=0x122794c0) at ../../src/qpid/sys/AsynchIOHandler.cpp:209
        #25 0x00002b0f04244aea in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0x48703ec0, a0=...) at /usr/include/boost/function/function_template.hpp:576
        #26 0x00002b0f0423eec6 in qpid::sys::posix::AsynchIO::writeable (this=0x120c0430, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:584
        #27 0x00002b0f0432590f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x48703ec0, a0=...)
            at /usr/include/boost/function/function_template.hpp:576
        #28 0x00002b0f04323d40 in qpid::sys::DispatchHandle::processEvent (this=0x120c0438, type=WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:283
        #29 0x00002b0f0424f708 in process (this=0x1209be70) at ../../src/qpid/sys/Poller.h:131
        #30 qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
        #31 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x100000001) at ../../src/qpid/sys/posix/Thread.cpp:35
        #32 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #33 0x00000039fead44bd in clone () from /lib64/libc.so.6
        (gdb) thread apply all bt
        
        Thread 18 (Thread 0x2b0f04e537b0 (LWP 32476)):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f03d2aea5 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:401
        #4  0x00000000004074ec in QpiddBroker::execute (this=<value optimized out>, options=0x120928f0) at ../../src/posix/QpiddBroker.cpp:195
        #5  0x0000000000405ab4 in run_broker (argc=10, argv=0x7fff5ec8b558, hidden=<value optimized out>) at ../../src/qpidd.cpp:83
        #6  0x00000039fea1d994 in __libc_start_main () from /lib64/libc.so.6
        #7  0x00000000004054e9 in _start ()
        
        Thread 17 (Thread 32492):
        #0  0x00000039ff60b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1  0x00002b0f0432b5bc in qpid::sys::Timer::run (this=0x1209c5e0) at ../../include/qpid/sys/posix/Condition.h:69
        #2  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x1209c614) at ../../src/qpid/sys/posix/Thread.cpp:35
        #3  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #4  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 16 (Thread 32493):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 15 (Thread 32494):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 14 (Thread 32495):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 13 (Thread 32496):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 12 (Thread 32497):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        ---Type <return> to continue, or q <return> to quit---
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 11 (Thread 32498):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 10 (Thread 32499):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 9 (Thread 32500):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 8 (Thread 32501):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 7 (Thread 32502):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 6 (Thread 32504):
        #0  0x00000039feac680b in write () from /lib64/libc.so.6
        #1  0x0000003a110bb8ee in ?? () from /usr/lib64/libstdc++.so.6
        #2  0x0000003a1106693e in std::basic_filebuf<char, std::char_traits<char> >::_M_convert_to_external(char*, long) () from /usr/lib64/libstdc++.so.6
        #3  0x0000003a11067be5 in std::basic_filebuf<char, std::char_traits<char> >::overflow(int) () from /usr/lib64/libstdc++.so.6
        #4  0x0000003a11066420 in std::basic_filebuf<char, std::char_traits<char> >::sync() () from /usr/lib64/libstdc++.so.6
        #5  0x0000003a1108f45b in std::basic_ostream<char, std::char_traits<char> >::flush() () from /usr/lib64/libstdc++.so.6
        #6  0x00002b0f04312da0 in operator() (__first=<value optimized out>, __last=..., __f=...) at /usr/include/boost/bind/mem_fn_template.hpp:262
        #7  operator()<boost::_mfi::mf2<void, qpid::log::Logger::Output, qpid::log::Statement const&, std::string const&>, boost::_bi::list1<qpid::log::Logger::Output&> > (__first=<value optimized out>, __last=..., 
            __f=...) at /usr/include/boost/bind.hpp:348
        ---Type <return> to continue, or q <return> to quit---
        #8  operator()<qpid::log::Logger::Output> (__first=<value optimized out>, __last=..., __f=...) at /usr/include/boost/bind/bind_template.hpp:32
        #9  std::for_each<boost::void_ptr_iterator<__gnu_cxx::__normal_iterator<void**, std::vector<void*, std::allocator<void*> > >, qpid::log::Logger::Output>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::log::Logger::Output, qpid::log::Statement const&, std::string const&>, boost::_bi::list3<boost::arg<1>, boost::_bi::value<qpid::log::Statement>, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > (__first=<value optimized out>, __last=..., __f=...) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_algo.h:159
        #10 0x00002b0f043113da in qpid::log::Logger::log (this=0x2b0f045c84e0, s=..., msg=
            "SENT [127.0.0.1:15672-127.0.0.1:60946]: Frame[Ebe; channel=0; content (1024 bytes) \\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\"...) at ../../src/qpid/log/Logger.cpp:115
        #11 0x00002b0f04316b5c in qpid::log::Statement::log (this=0x2b0f04104480, message="SENT [127.0.0.1:15672-127.0.0.1:60946]: Frame[Ebe; channel=0; content (1024 bytes) ", '\000' <repeats 32 times>, "...]")
            at ../../src/qpid/log/Statement.cpp:54
        #12 0x00002b0f03d1abbd in qpid::amqp_0_10::Connection::encode (this=0x120bdab0, buffer=<value optimized out>, size=<value optimized out>) at ../../src/qpid/amqp_0_10/Connection.cpp:95
        #13 0x00002b0f043204ae in qpid::sys::AsynchIOHandler::idle (this=0x120b5ee0) at ../../src/qpid/sys/AsynchIOHandler.cpp:213
        #14 0x00002b0f04244aea in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0xf2, a0=...) at /usr/include/boost/function/function_template.hpp:576
        #15 0x00002b0f0423eec6 in qpid::sys::posix::AsynchIO::writeable (this=0x120c0040, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:584
        #16 0x00002b0f0432590f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xf2, a0=...) at /usr/include/boost/function/function_template.hpp:576
        #17 0x00002b0f04323d40 in qpid::sys::DispatchHandle::processEvent (this=0x120c0048, type=WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:283
        #18 0x00002b0f0424f708 in process (this=0x1209be70) at ../../src/qpid/sys/Poller.h:131
        #19 qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
        #20 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35
        #21 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #22 0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 5 (Thread 32505):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 4 (Thread 32506):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 3 (Thread 32507):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 2 (Thread 32508):
        #0  0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6
        #1  0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568
        #2  0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520
        #3  0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35
        #4  0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #5  0x00000039fead44bd in clone () from /lib64/libc.so.6
        
        Thread 1 (Thread 0x48705940 (LWP 32503)):
        ---Type <return> to continue, or q <return> to quit---
        #0  qpid::framing::FieldValue::print (this=0x2aaab003acc0, out=...) at ../../src/qpid/framing/FieldValue.cpp:223
        #1  0x00002b0f042fec09 in operator<< (out=<value optimized out>, i=<value optimized out>) at ../../include/qpid/framing/FieldValue.h:136
        #2  qpid::framing::(anonymous namespace)::operator<< (out=<value optimized out>, i=<value optimized out>) at ../../src/qpid/framing/FieldTable.cpp:78
        #3  0x00002b0f04300d3b in qpid::framing::operator<< (out=..., t=...) at ../../src/qpid/framing/FieldTable.cpp:86
        #4  0x00002b0f04277bc0 in qpid::framing::MessageProperties::print (this=0x2aaab004dd78, out=...) at qpid/framing/MessageProperties.cpp:221
        #5  0x00002b0f04277daf in qpid::framing::operator<< (out=..., s=...) at qpid/framing/MessageProperties.cpp:229
        #6  0x00002b0f042f218e in print (this=0x2aaab004dd10, out=...) at ../../src/qpid/framing/AMQHeaderBody.h:64
        #7  qpid::framing::AMQHeaderBody::print (this=0x2aaab004dd10, out=...) at ../../src/qpid/framing/AMQHeaderBody.cpp:57
        #8  0x00002b0f042f0360 in qpid::framing::operator<< (out=..., body=...) at ../../src/qpid/framing/AMQBody.cpp:34
        #9  0x00002b0f042e2e5d in operator<< <qpid::framing::AMQBody> (this=0x2aaab00582e0, f=...) at ../../include/qpid/Msg.h:63
        #10 qpid::SessionState::senderRecord (this=0x2aaab00582e0, f=...) at ../../src/qpid/SessionState.cpp:119
        #11 0x00002b0f042ebc85 in qpid::amqp_0_10::SessionHandler::handleOut (this=0x2aaab0105a10, f=...) at ../../src/qpid/amqp_0_10/SessionHandler.cpp:142
        #12 0x00002b0f03d9b720 in map_if<qpid::framing::Relay, qpid::framing::TypeFilter<2u> > (this=0x2aaab81325e0, out=...) at ../../src/qpid/framing/frame_functors.h:90
        #13 qpid::broker::Message::sendHeader (this=0x2aaab81325e0, out=...) at ../../src/qpid/broker/Message.cpp:291
        #14 0x00002b0f03d5a8a3 in qpid::broker::DeliveryRecord::deliver (this=0x487043f0, h=..., deliveryId=<value optimized out>, framesize=65535) at ../../src/qpid/broker/DeliveryRecord.cpp:89
        #15 0x00002b0f03e1c7d4 in qpid::broker::SessionState::deliver (this=0x2aaab00582e0, msg=..., sync=false) at ../../src/qpid/broker/SessionState.cpp:302
        #16 0x00002b0f03df9e70 in qpid::broker::SemanticState::ConsumerImpl::deliver (this=0x2aaab8136500, msg=...) at ../../src/qpid/broker/SemanticState.cpp:348
        #17 0x00002b0f03dbf2e6 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at ../../src/qpid/broker/Queue.cpp:475
        #18 0x00002b0f03df7dcc in qpid::broker::SemanticState::ConsumerImpl::doDispatch (this=<value optimized out>) at ../../src/qpid/broker/SemanticState.cpp:664
        #19 0x00002b0f03df71f9 in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x2aaab8136500) at ../../src/qpid/broker/SemanticState.cpp:731
        #20 0x00002b0f0431cdf3 in qpid::sys::AggregateOutput::doOutput (this=<value optimized out>) at ../../src/qpid/sys/AggregateOutput.cpp:59
        #21 0x00002b0f03d4b509 in qpid::broker::Connection::doOutput (this=0x2aaab005a3e0) at ../../src/qpid/broker/Connection.cpp:360
        #22 0x00002b0f03d19ca0 in qpid::amqp_0_10::Connection::canEncode (this=0x2aaab0104240) at ../../src/qpid/amqp_0_10/Connection.cpp:67
        #23 0x00002b0f03df61ba in qpid::broker::SecureConnection::canEncode (this=<value optimized out>) at ../../src/qpid/broker/SecureConnection.cpp:58
        #24 0x00002b0f04320480 in qpid::sys::AsynchIOHandler::idle (this=0x122794c0) at ../../src/qpid/sys/AsynchIOHandler.cpp:209
        #25 0x00002b0f04244aea in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0x48703ec0, a0=...) at /usr/include/boost/function/function_template.hpp:576
        #26 0x00002b0f0423eec6 in qpid::sys::posix::AsynchIO::writeable (this=0x120c0430, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:584
        #27 0x00002b0f0432590f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x48703ec0, a0=...)
            at /usr/include/boost/function/function_template.hpp:576
        #28 0x00002b0f04323d40 in qpid::sys::DispatchHandle::processEvent (this=0x120c0438, type=WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:283
        #29 0x00002b0f0424f708 in process (this=0x1209be70) at ../../src/qpid/sys/Poller.h:131
        #30 qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524
        #31 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x100000001) at ../../src/qpid/sys/posix/Thread.cpp:35
        #32 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0
        #33 0x00000039fead44bd in clone () from /lib64/libc.so.6
        (gdb) 
        
        Show
        Gordon Sim added a comment - Program terminated with signal 11, Segmentation fault. #0 qpid::framing::FieldValue::print (this=0x2aaab003acc0, out=...) at ../../src/qpid/framing/FieldValue.cpp:223 223 if (data->convertsToString()) out << data->getString(); (gdb) bt #0 qpid::framing::FieldValue::print (this=0x2aaab003acc0, out=...) at ../../src/qpid/framing/FieldValue.cpp:223 #1 0x00002b0f042fec09 in operator<< (out=<value optimized out>, i=<value optimized out>) at ../../include/qpid/framing/FieldValue.h:136 #2 qpid::framing::(anonymous namespace)::operator<< (out=<value optimized out>, i=<value optimized out>) at ../../src/qpid/framing/FieldTable.cpp:78 #3 0x00002b0f04300d3b in qpid::framing::operator<< (out=..., t=...) at ../../src/qpid/framing/FieldTable.cpp:86 #4 0x00002b0f04277bc0 in qpid::framing::MessageProperties::print (this=0x2aaab004dd78, out=...) at qpid/framing/MessageProperties.cpp:221 #5 0x00002b0f04277daf in qpid::framing::operator<< (out=..., s=...) at qpid/framing/MessageProperties.cpp:229 #6 0x00002b0f042f218e in print (this=0x2aaab004dd10, out=...) at ../../src/qpid/framing/AMQHeaderBody.h:64 #7 qpid::framing::AMQHeaderBody::print (this=0x2aaab004dd10, out=...) at ../../src/qpid/framing/AMQHeaderBody.cpp:57 #8 0x00002b0f042f0360 in qpid::framing::operator<< (out=..., body=...) at ../../src/qpid/framing/AMQBody.cpp:34 #9 0x00002b0f042e2e5d in operator<< <qpid::framing::AMQBody> (this=0x2aaab00582e0, f=...) at ../../include/qpid/Msg.h:63 #10 qpid::SessionState::senderRecord (this=0x2aaab00582e0, f=...) at ../../src/qpid/SessionState.cpp:119 #11 0x00002b0f042ebc85 in qpid::amqp_0_10::SessionHandler::handleOut (this=0x2aaab0105a10, f=...) at ../../src/qpid/amqp_0_10/SessionHandler.cpp:142 #12 0x00002b0f03d9b720 in map_if<qpid::framing::Relay, qpid::framing::TypeFilter<2u> > (this=0x2aaab81325e0, out=...) at ../../src/qpid/framing/frame_functors.h:90 #13 qpid::broker::Message::sendHeader (this=0x2aaab81325e0, out=...) at ../../src/qpid/broker/Message.cpp:291 #14 0x00002b0f03d5a8a3 in qpid::broker::DeliveryRecord::deliver (this=0x487043f0, h=..., deliveryId=<value optimized out>, framesize=65535) at ../../src/qpid/broker/DeliveryRecord.cpp:89 #15 0x00002b0f03e1c7d4 in qpid::broker::SessionState::deliver (this=0x2aaab00582e0, msg=..., sync=false) at ../../src/qpid/broker/SessionState.cpp:302 #16 0x00002b0f03df9e70 in qpid::broker::SemanticState::ConsumerImpl::deliver (this=0x2aaab8136500, msg=...) at ../../src/qpid/broker/SemanticState.cpp:348 #17 0x00002b0f03dbf2e6 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at ../../src/qpid/broker/Queue.cpp:475 #18 0x00002b0f03df7dcc in qpid::broker::SemanticState::ConsumerImpl::doDispatch (this=<value optimized out>) at ../../src/qpid/broker/SemanticState.cpp:664 #19 0x00002b0f03df71f9 in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x2aaab8136500) at ../../src/qpid/broker/SemanticState.cpp:731 #20 0x00002b0f0431cdf3 in qpid::sys::AggregateOutput::doOutput (this=<value optimized out>) at ../../src/qpid/sys/AggregateOutput.cpp:59 #21 0x00002b0f03d4b509 in qpid::broker::Connection::doOutput (this=0x2aaab005a3e0) at ../../src/qpid/broker/Connection.cpp:360 #22 0x00002b0f03d19ca0 in qpid::amqp_0_10::Connection::canEncode (this=0x2aaab0104240) at ../../src/qpid/amqp_0_10/Connection.cpp:67 #23 0x00002b0f03df61ba in qpid::broker::SecureConnection::canEncode (this=<value optimized out>) at ../../src/qpid/broker/SecureConnection.cpp:58 #24 0x00002b0f04320480 in qpid::sys::AsynchIOHandler::idle (this=0x122794c0) at ../../src/qpid/sys/AsynchIOHandler.cpp:209 #25 0x00002b0f04244aea in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0x48703ec0, a0=...) at /usr/include/boost/function/function_template.hpp:576 #26 0x00002b0f0423eec6 in qpid::sys::posix::AsynchIO::writeable (this=0x120c0430, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:584 #27 0x00002b0f0432590f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x48703ec0, a0=...) at /usr/include/boost/function/function_template.hpp:576 #28 0x00002b0f04323d40 in qpid::sys::DispatchHandle::processEvent (this=0x120c0438, type=WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:283 #29 0x00002b0f0424f708 in process (this=0x1209be70) at ../../src/qpid/sys/Poller.h:131 #30 qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #31 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x100000001) at ../../src/qpid/sys/posix/Thread.cpp:35 #32 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #33 0x00000039fead44bd in clone () from /lib64/libc.so.6 (gdb) thread apply all bt Thread 18 (Thread 0x2b0f04e537b0 (LWP 32476)): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f03d2aea5 in qpid::broker::Broker::run (this=<value optimized out>) at ../../src/qpid/broker/Broker.cpp:401 #4 0x00000000004074ec in QpiddBroker::execute (this=<value optimized out>, options=0x120928f0) at ../../src/posix/QpiddBroker.cpp:195 #5 0x0000000000405ab4 in run_broker (argc=10, argv=0x7fff5ec8b558, hidden=<value optimized out>) at ../../src/qpidd.cpp:83 #6 0x00000039fea1d994 in __libc_start_main () from /lib64/libc.so.6 #7 0x00000000004054e9 in _start () Thread 17 (Thread 32492): #0 0x00000039ff60b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b0f0432b5bc in qpid::sys::Timer::run (this=0x1209c5e0) at ../../include/qpid/sys/posix/Condition.h:69 #2 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x1209c614) at ../../src/qpid/sys/posix/Thread.cpp:35 #3 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #4 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 16 (Thread 32493): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 15 (Thread 32494): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 14 (Thread 32495): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 13 (Thread 32496): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 12 (Thread 32497): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 ---Type <return> to continue, or q <return> to quit--- #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 11 (Thread 32498): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 10 (Thread 32499): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 9 (Thread 32500): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 8 (Thread 32501): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 7 (Thread 32502): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 6 (Thread 32504): #0 0x00000039feac680b in write () from /lib64/libc.so.6 #1 0x0000003a110bb8ee in ?? () from /usr/lib64/libstdc++.so.6 #2 0x0000003a1106693e in std::basic_filebuf<char, std::char_traits<char> >::_M_convert_to_external(char*, long) () from /usr/lib64/libstdc++.so.6 #3 0x0000003a11067be5 in std::basic_filebuf<char, std::char_traits<char> >::overflow(int) () from /usr/lib64/libstdc++.so.6 #4 0x0000003a11066420 in std::basic_filebuf<char, std::char_traits<char> >::sync() () from /usr/lib64/libstdc++.so.6 #5 0x0000003a1108f45b in std::basic_ostream<char, std::char_traits<char> >::flush() () from /usr/lib64/libstdc++.so.6 #6 0x00002b0f04312da0 in operator() (__first=<value optimized out>, __last=..., __f=...) at /usr/include/boost/bind/mem_fn_template.hpp:262 #7 operator()<boost::_mfi::mf2<void, qpid::log::Logger::Output, qpid::log::Statement const&, std::string const&>, boost::_bi::list1<qpid::log::Logger::Output&> > (__first=<value optimized out>, __last=..., __f=...) at /usr/include/boost/bind.hpp:348 ---Type <return> to continue, or q <return> to quit--- #8 operator()<qpid::log::Logger::Output> (__first=<value optimized out>, __last=..., __f=...) at /usr/include/boost/bind/bind_template.hpp:32 #9 std::for_each<boost::void_ptr_iterator<__gnu_cxx::__normal_iterator<void**, std::vector<void*, std::allocator<void*> > >, qpid::log::Logger::Output>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::log::Logger::Output, qpid::log::Statement const&, std::string const&>, boost::_bi::list3<boost::arg<1>, boost::_bi::value<qpid::log::Statement>, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > (__first=<value optimized out>, __last=..., __f=...) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_algo.h:159 #10 0x00002b0f043113da in qpid::log::Logger::log (this=0x2b0f045c84e0, s=..., msg= "SENT [127.0.0.1:15672-127.0.0.1:60946]: Frame[Ebe; channel=0; content (1024 bytes) \\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\"...) at ../../src/qpid/log/Logger.cpp:115 #11 0x00002b0f04316b5c in qpid::log::Statement::log (this=0x2b0f04104480, message="SENT [127.0.0.1:15672-127.0.0.1:60946]: Frame[Ebe; channel=0; content (1024 bytes) ", '\000' <repeats 32 times>, "...]") at ../../src/qpid/log/Statement.cpp:54 #12 0x00002b0f03d1abbd in qpid::amqp_0_10::Connection::encode (this=0x120bdab0, buffer=<value optimized out>, size=<value optimized out>) at ../../src/qpid/amqp_0_10/Connection.cpp:95 #13 0x00002b0f043204ae in qpid::sys::AsynchIOHandler::idle (this=0x120b5ee0) at ../../src/qpid/sys/AsynchIOHandler.cpp:213 #14 0x00002b0f04244aea in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0xf2, a0=...) at /usr/include/boost/function/function_template.hpp:576 #15 0x00002b0f0423eec6 in qpid::sys::posix::AsynchIO::writeable (this=0x120c0040, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:584 #16 0x00002b0f0432590f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xf2, a0=...) at /usr/include/boost/function/function_template.hpp:576 #17 0x00002b0f04323d40 in qpid::sys::DispatchHandle::processEvent (this=0x120c0048, type=WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:283 #18 0x00002b0f0424f708 in process (this=0x1209be70) at ../../src/qpid/sys/Poller.h:131 #19 qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #20 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x5) at ../../src/qpid/sys/posix/Thread.cpp:35 #21 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #22 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 5 (Thread 32505): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 4 (Thread 32506): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 3 (Thread 32507): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 2 (Thread 32508): #0 0x00000039fead48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b0f0424e97b in qpid::sys::Poller::wait (this=0x1209be70, timeout=<value optimized out>) at ../../src/qpid/sys/epoll/EpollPoller.cpp:568 #2 0x00002b0f0424f717 in qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:520 #3 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x6) at ../../src/qpid/sys/posix/Thread.cpp:35 #4 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000039fead44bd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x48705940 (LWP 32503)): ---Type <return> to continue, or q <return> to quit--- #0 qpid::framing::FieldValue::print (this=0x2aaab003acc0, out=...) at ../../src/qpid/framing/FieldValue.cpp:223 #1 0x00002b0f042fec09 in operator<< (out=<value optimized out>, i=<value optimized out>) at ../../include/qpid/framing/FieldValue.h:136 #2 qpid::framing::(anonymous namespace)::operator<< (out=<value optimized out>, i=<value optimized out>) at ../../src/qpid/framing/FieldTable.cpp:78 #3 0x00002b0f04300d3b in qpid::framing::operator<< (out=..., t=...) at ../../src/qpid/framing/FieldTable.cpp:86 #4 0x00002b0f04277bc0 in qpid::framing::MessageProperties::print (this=0x2aaab004dd78, out=...) at qpid/framing/MessageProperties.cpp:221 #5 0x00002b0f04277daf in qpid::framing::operator<< (out=..., s=...) at qpid/framing/MessageProperties.cpp:229 #6 0x00002b0f042f218e in print (this=0x2aaab004dd10, out=...) at ../../src/qpid/framing/AMQHeaderBody.h:64 #7 qpid::framing::AMQHeaderBody::print (this=0x2aaab004dd10, out=...) at ../../src/qpid/framing/AMQHeaderBody.cpp:57 #8 0x00002b0f042f0360 in qpid::framing::operator<< (out=..., body=...) at ../../src/qpid/framing/AMQBody.cpp:34 #9 0x00002b0f042e2e5d in operator<< <qpid::framing::AMQBody> (this=0x2aaab00582e0, f=...) at ../../include/qpid/Msg.h:63 #10 qpid::SessionState::senderRecord (this=0x2aaab00582e0, f=...) at ../../src/qpid/SessionState.cpp:119 #11 0x00002b0f042ebc85 in qpid::amqp_0_10::SessionHandler::handleOut (this=0x2aaab0105a10, f=...) at ../../src/qpid/amqp_0_10/SessionHandler.cpp:142 #12 0x00002b0f03d9b720 in map_if<qpid::framing::Relay, qpid::framing::TypeFilter<2u> > (this=0x2aaab81325e0, out=...) at ../../src/qpid/framing/frame_functors.h:90 #13 qpid::broker::Message::sendHeader (this=0x2aaab81325e0, out=...) at ../../src/qpid/broker/Message.cpp:291 #14 0x00002b0f03d5a8a3 in qpid::broker::DeliveryRecord::deliver (this=0x487043f0, h=..., deliveryId=<value optimized out>, framesize=65535) at ../../src/qpid/broker/DeliveryRecord.cpp:89 #15 0x00002b0f03e1c7d4 in qpid::broker::SessionState::deliver (this=0x2aaab00582e0, msg=..., sync=false) at ../../src/qpid/broker/SessionState.cpp:302 #16 0x00002b0f03df9e70 in qpid::broker::SemanticState::ConsumerImpl::deliver (this=0x2aaab8136500, msg=...) at ../../src/qpid/broker/SemanticState.cpp:348 #17 0x00002b0f03dbf2e6 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at ../../src/qpid/broker/Queue.cpp:475 #18 0x00002b0f03df7dcc in qpid::broker::SemanticState::ConsumerImpl::doDispatch (this=<value optimized out>) at ../../src/qpid/broker/SemanticState.cpp:664 #19 0x00002b0f03df71f9 in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x2aaab8136500) at ../../src/qpid/broker/SemanticState.cpp:731 #20 0x00002b0f0431cdf3 in qpid::sys::AggregateOutput::doOutput (this=<value optimized out>) at ../../src/qpid/sys/AggregateOutput.cpp:59 #21 0x00002b0f03d4b509 in qpid::broker::Connection::doOutput (this=0x2aaab005a3e0) at ../../src/qpid/broker/Connection.cpp:360 #22 0x00002b0f03d19ca0 in qpid::amqp_0_10::Connection::canEncode (this=0x2aaab0104240) at ../../src/qpid/amqp_0_10/Connection.cpp:67 #23 0x00002b0f03df61ba in qpid::broker::SecureConnection::canEncode (this=<value optimized out>) at ../../src/qpid/broker/SecureConnection.cpp:58 #24 0x00002b0f04320480 in qpid::sys::AsynchIOHandler::idle (this=0x122794c0) at ../../src/qpid/sys/AsynchIOHandler.cpp:209 #25 0x00002b0f04244aea in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0x48703ec0, a0=...) at /usr/include/boost/function/function_template.hpp:576 #26 0x00002b0f0423eec6 in qpid::sys::posix::AsynchIO::writeable (this=0x120c0430, h=...) at ../../src/qpid/sys/posix/AsynchIO.cpp:584 #27 0x00002b0f0432590f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x48703ec0, a0=...) at /usr/include/boost/function/function_template.hpp:576 #28 0x00002b0f04323d40 in qpid::sys::DispatchHandle::processEvent (this=0x120c0438, type=WRITABLE) at ../../src/qpid/sys/DispatchHandle.cpp:283 #29 0x00002b0f0424f708 in process (this=0x1209be70) at ../../src/qpid/sys/Poller.h:131 #30 qpid::sys::Poller::run (this=0x1209be70) at ../../src/qpid/sys/epoll/EpollPoller.cpp:524 #31 0x00002b0f0424709a in qpid::sys::(anonymous namespace)::runRunnable (p=0x100000001) at ../../src/qpid/sys/posix/Thread.cpp:35 #32 0x00000039ff60673d in start_thread () from /lib64/libpthread.so.0 #33 0x00000039fead44bd in clone () from /lib64/libc.so.6 (gdb)
        Robbie Gemmell made changes -
        Affects Version/s 0.16 [ 12319870 ]
        Affects Version/s 0.17 [ 12320179 ]
        Hide
        Robbie Gemmell added a comment -

        Setting affects versions as 0.15, 0.16, 0.17 for now.

        Show
        Robbie Gemmell added a comment - Setting affects versions as 0.15, 0.16, 0.17 for now.
        Robbie Gemmell made changes -
        Attachment testlog.zip [ 12517907 ]
        Hide
        Robbie Gemmell added a comment -

        Attaching test log output from build #496

        Show
        Robbie Gemmell added a comment - Attaching test log output from build #496
        Robbie Gemmell made changes -
        Field Original Value New Value
        Description The C++ broker appears to segfault during MultipleTransactedBatchProducerTest. Several instances can be found in the history from our CI jobs:

        https://builds.apache.org/job/Qpid-Java-Cpp-Test/496/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/494/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/493/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/491/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/486/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/484/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/482/

        During course of the latest test failure in #496 (logs attached), the broker logged receiving yet another TxCommit command:

        {noformat}
        BROKER: 2012-03-11 16:55:15 trace RECV [127.0.0.1:15672-127.0.0.1:34582]: Frame[BEbe; channel=0; {TxCommitBody: }] )
        {noformat}

        After this nothing else is ever logged from the broker process, but exceptions start flying from the client indicating the connections were reset:

        {noformat}
        IoReceiver - localhost/127.0.0.1:15672 2012-03-11 16:55:15,313 ERROR [apache.qpid.client.AMQConnectionDelegate_0_10] previous exception
        org.apache.qpid.transport.ConnectionException: Connection reset
        at org.apache.qpid.transport.Connection.exception(Connection.java:512)
        at org.apache.qpid.transport.network.Assembler.exception(Assembler.java:107)
        at org.apache.qpid.transport.network.InputHandler.exception(InputHandler.java:199)
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:169)
        at java.lang.Thread.run(Thread.java:662)
        Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:147)
        ... 1 more
        {noformat}

        {noformat}
        IoSender - localhost/127.0.0.1:15672 2012-03-11 16:55:15,313 ERROR [transport.network.io.IoSender] error in write thread
        java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:313)
        at java.lang.Thread.run(Thread.java:662)
        {noformat}

        When the test tearDown occurs, it is logged that the broker process exited with an abnormal looking exit code, which a quick google suggests is the result of a segfault:
        {noformat}
        main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.QpidBrokerTestCase] stopping broker on port : 15672
        main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.SpawnedBrokerHolder] Destroying broker process
        main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.SpawnedBrokerHolder] broker exited: 139
        {noformat}

        The test fails after 75sec, which is the exact time the test itself allows for the process undertaken to complete (artificially high to allow for extreme slowdowns on the CI nodes which happened occasionally in the past and also the C++ broker being slower at the test due to use of client side selectors; passing results show this usually takes 1-3sec on the Java broker and <10sec on the [transient] C++ broker), however the test hardness tries to clean up the connections use in the test during tearDown and actually ends up reporting issues cleaning those up rather than the fact that the CountDownLatch used in the test wasnt decremented enough times.
        The C++ broker appears to segfault during MultipleTransactedBatchProducerTest. Several instances can be found in the history from our CI jobs:

        https://builds.apache.org/job/Qpid-Java-Cpp-Test/496/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/494/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/493/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/491/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/486/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/484/
        https://builds.apache.org/job/Qpid-Java-Cpp-Test/482/

        During course of the latest test failure in #496 (logs attached), the broker logged receiving yet another TxCommit command:

        {noformat}
        BROKER: 2012-03-11 16:55:15 trace RECV [127.0.0.1:15672-127.0.0.1:34582]: Frame[BEbe; channel=0; {TxCommitBody: }] )
        {noformat}

        After this nothing else is ever logged from the broker process, but exceptions start flying from the client indicating the connections were reset:

        {noformat}
        IoReceiver - localhost/127.0.0.1:15672 2012-03-11 16:55:15,313 ERROR [apache.qpid.client.AMQConnectionDelegate_0_10] previous exception
        org.apache.qpid.transport.ConnectionException: Connection reset
        at org.apache.qpid.transport.Connection.exception(Connection.java:512)
        at org.apache.qpid.transport.network.Assembler.exception(Assembler.java:107)
        at org.apache.qpid.transport.network.InputHandler.exception(InputHandler.java:199)
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:169)
        at java.lang.Thread.run(Thread.java:662)
        Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:147)
        ... 1 more
        {noformat}

        {noformat}
        IoSender - localhost/127.0.0.1:15672 2012-03-11 16:55:15,313 ERROR [transport.network.io.IoSender] error in write thread
        java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:313)
        at java.lang.Thread.run(Thread.java:662)
        {noformat}

        When the test tearDown occurs, it is logged that the broker process exited with an abnormal looking exit code, which a quick google suggests is the result of a segfault:
        {noformat}
        main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.QpidBrokerTestCase] stopping broker on port : 15672
        main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.SpawnedBrokerHolder] Destroying broker process
        main 2012-03-11 16:56:29,949 INFO [qpid.test.utils.SpawnedBrokerHolder] broker exited: 139
        {noformat}

        The test fails after 75sec, which is the exact time the test itself allows for the process undertaken to complete (artificially high to allow for extreme slowdowns on the CI nodes which happened occasionally in the past and also the C++ broker being slower at the test due to use of client side selectors; passing results show this usually takes 1-3sec on the Java broker and <10sec on the [transient] C++ broker), however the test harness tries to clean up the connections use in the test during tearDown and actually ends up reporting issues cleaning those up rather than the fact that the CountDownLatch used in the test wasnt decremented enough times.
        Robbie Gemmell created issue -

          People

          • Assignee:
            Andrew Stitcher
            Reporter:
            Robbie Gemmell
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development