Uploaded image for project: 'Qpid Dispatch'
  1. Qpid Dispatch
  2. DISPATCH-1285

Router crashes occasionally on system_tests_delivery_abort

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.6.0
    • Component/s: Container
    • Labels:
      None

      Description

      The router occasionally crashes with the following backtrace when running system_tests_delivery_abort. Dont run the test independently, run it as part of the test suite using ctest -VV

       

      (gdb) bt
      #0  0x00007f3f763d8aaf in qd_link_get_node_context (link=0x7f3f580b40e8) at /home/gmurthy/opensource/qpid-dispatch/src/container.c:1053
      #1  0x00007f3f7642595f in deferred_AMQP_rx_handler (context=0x7f3f580b40e8, discard=false) at /home/gmurthy/opensource/qpid-dispatch/src/router_node.c:640
      #2  0x00007f3f7642bf02 in invoke_deferred_calls (conn=0x1b5a568, discard=false) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:728
      #3  0x00007f3f7642c9c2 in handle (qd_server=0x1a0f5b0, e=0x7f3f58093520, pn_conn=0x1bf4e40, ctx=0x1b5a568) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:961
      #4  0x00007f3f7642cd2a in thread_run (arg=0x1a0f5b0) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:1017
      #5  0x00007f3f7632258e in start_thread () from /lib64/libpthread.so.0
      #6  0x00007f3f75dca6a3 in clone () from /lib64/libc.so.6
      (gdb)

       

      Here is the output from the test -

      40: Test command: /usr/bin/python "/home/gmurthy/opensource/qpid-dispatch/build/tests/run.py" "unit2" "-v" "system_tests_delivery_abort"
      40: Test timeout computed to be: 1500
      40: test_01_message_route_truncated_one_router (system_tests_delivery_abort.RouterTest) ... ok
      40: test_02_message_route_truncated_two_routers (system_tests_delivery_abort.RouterTest) ... ok
      40: test_03_link_route_truncated_one_router (system_tests_delivery_abort.RouterTest) ... ok
      40: test_04_link_route_truncated_two_routers (system_tests_delivery_abort.RouterTest) ... ERROR
      40: test_05_message_route_abort_one_router (system_tests_delivery_abort.RouterTest) ... ERROR
      40: test_06_message_route_abort_two_routers (system_tests_delivery_abort.RouterTest) ... ERROR
      40: test_07_multicast_truncate_one_router (system_tests_delivery_abort.RouterTest) ... ERROR
      40: ERROR
      40:
      40: ======================================================================
      40: ERROR: test_04_link_route_truncated_two_routers (system_tests_delivery_abort.RouterTest)
      40: ----------------------------------------------------------------------
      40: Traceback (most recent call last):
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_delivery_abort.py", line 101, in test_04_link_route_truncated_two_routers
      40:     test.run()
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_delivery_abort.py", line 390, in run
      40:     container.run()
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 181, in run
      40:     while self.process(): pass
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 240, in process
      40:     event.dispatch(self._global_handler)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 135, in dispatch
      40:     _dispatch(handler, type.method, self)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 117, in _dispatch
      40:     handler.on_unhandled(method, *args)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 665, in on_unhandled
      40:     event.dispatch(self.base)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 135, in dispatch
      40:     _dispatch(handler, type.method, self)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 115, in _dispatch
      40:     m(*args)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_handlers.py", line 888, in on_connection_bound
      40:     sock = IO.connect(host, int(port))
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_io.py", line 48, in connect
      40:     return socket.create_connection((host, port))
      40:   File "/usr/lib64/python2.7/socket.py", line 575, in create_connection
      40:     raise err
      40: socket.error: [Errno 111] Connection refused
      40:
      40: ======================================================================
      40: ERROR: test_05_message_route_abort_one_router (system_tests_delivery_abort.RouterTest)
      40: ----------------------------------------------------------------------
      40: Traceback (most recent call last):
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_delivery_abort.py", line 109, in test_05_message_route_abort_one_router
      40:     test.run()
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_delivery_abort.py", line 473, in run
      40:     Container(self).run()
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 181, in run
      40:     while self.process(): pass
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 240, in process
      40:     event.dispatch(self._global_handler)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 135, in dispatch
      40:     _dispatch(handler, type.method, self)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 117, in _dispatch
      40:     handler.on_unhandled(method, *args)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 665, in on_unhandled
      40:     event.dispatch(self.base)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 135, in dispatch
      40:     _dispatch(handler, type.method, self)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 115, in _dispatch
      40:     m(*args)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_handlers.py", line 888, in on_connection_bound
      40:     sock = IO.connect(host, int(port))
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_io.py", line 48, in connect
      40:     return socket.create_connection((host, port))
      40:   File "/usr/lib64/python2.7/socket.py", line 575, in create_connection
      40:     raise err
      40: socket.error: [Errno 111] Connection refused
      40:
      40: ======================================================================
      40: ERROR: test_06_message_route_abort_two_routers (system_tests_delivery_abort.RouterTest)
      40: ----------------------------------------------------------------------
      40: Traceback (most recent call last):
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_delivery_abort.py", line 117, in test_06_message_route_abort_two_routers
      40:     test.run()
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_delivery_abort.py", line 473, in run
      40:     Container(self).run()
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 181, in run
      40:     while self.process(): pass
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 240, in process
      40:     event.dispatch(self._global_handler)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 135, in dispatch
      40:     _dispatch(handler, type.method, self)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 117, in _dispatch
      40:     handler.on_unhandled(method, *args)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 665, in on_unhandled
      40:     event.dispatch(self.base)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 135, in dispatch
      40:     _dispatch(handler, type.method, self)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 115, in _dispatch
      40:     m(*args)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_handlers.py", line 888, in on_connection_bound
      40:     sock = IO.connect(host, int(port))
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_io.py", line 48, in connect
      40:     return socket.create_connection((host, port))
      40:   File "/usr/lib64/python2.7/socket.py", line 575, in create_connection
      40:     raise err
      40: socket.error: [Errno 111] Connection refused
      40:
      40: ======================================================================
      40: ERROR: test_07_multicast_truncate_one_router (system_tests_delivery_abort.RouterTest)
      40: ----------------------------------------------------------------------
      40: Traceback (most recent call last):
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_delivery_abort.py", line 126, in test_07_multicast_truncate_one_router
      40:     test.run()
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_tests_delivery_abort.py", line 591, in run
      40:     Container(self).run()
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 181, in run
      40:     while self.process(): pass
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 240, in process
      40:     event.dispatch(self._global_handler)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 135, in dispatch
      40:     _dispatch(handler, type.method, self)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 117, in _dispatch
      40:     handler.on_unhandled(method, *args)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_reactor.py", line 665, in on_unhandled
      40:     event.dispatch(self.base)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 135, in dispatch
      40:     _dispatch(handler, type.method, self)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_events.py", line 115, in _dispatch
      40:     m(*args)
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_handlers.py", line 888, in on_connection_bound
      40:     sock = IO.connect(host, int(port))
      40:   File "/opt/qpid-proton/qpid-proton/lib64/proton/bindings/python/proton/_io.py", line 48, in connect
      40:     return socket.create_connection((host, port))
      40:   File "/usr/lib64/python2.7/socket.py", line 575, in create_connection
      40:     raise err
      40: socket.error: [Errno 111] Connection refused
      40:
      40: ======================================================================
      40: ERROR: tearDownClass (system_tests_delivery_abort.RouterTest)
      40: ----------------------------------------------------------------------
      40: Traceback (most recent call last):
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_test.py", line 609, in tearDownClass
      40:     cls.tester.teardown()
      40:   File "/home/gmurthy/opensource/qpid-dispatch/tests/system_test.py", line 555, in teardown
      40:     raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
      40: RuntimeError: Errors during teardown:
      40:
      40: Process 6811 error: exit code -11, expected -1
      40: qdrouterd -c A.conf -I /home/gmurthy/opensource/qpid-dispatch/python
      40: /home/gmurthy/opensource/qpid-dispatch/build/tests/system_test.dir/system_tests_delivery_abort/RouterTest/setUpClass/A-1.cmd
      40: >>>>
      40: <<<<
      

       

       

        Attachments

          Activity

            People

            • Assignee:
              ganeshmurthy Ganesh Murthy
              Reporter:
              ganeshmurthy Ganesh Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: