Description
The router crashes during shutdown in system_tests_distribution.py
Here is the backtrace
(gdb) bt #0 0x00007f361ca5ae40 in pn_ep_decref (endpoint=0x7f35f01c2dd0) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:447 #1 0x00007f361ca5b58b in pn_ep_decref (endpoint=<optimized out>) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/engine.c:445 #2 0x00007f361ca5f588 in pni_transport_unbind_handles (handles=0x7f35f00764a0, reset_state=reset_state@entry=true) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:748 #3 0x00007f361ca5f666 in pni_transport_unbind_channels (channels=0x9d1ce0) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:761 #4 0x00007f361ca5f777 in pn_transport_unbind (transport=0xa863d0) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/transport.c:795 #5 0x00007f361ca5a63e in pn_connection_driver_release_connection (d=d@entry=0xa86248) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:81 #6 0x00007f361ca5a679 in pn_connection_driver_destroy (d=d@entry=0xa86248) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/core/connection_driver.c:92 #7 0x00007f361c83a69c in pconnection_final_free (pc=0xa85ca0) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:827 #8 0x00007f361c83b3ac in pconnection_cleanup (pc=<optimized out>) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:843 #9 0x00007f361c83db37 in pconnection_forced_shutdown (pc=0xa85ca0) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:878 #10 pn_proactor_free (p=0x916fd0) at /home/gmurthy/opensource/qpid-proton-0.22.0/proton-c/src/proactor/epoll.c:1815 #11 0x00007f361cce7bb5 in qd_server_free (qd_server=0x919190) at /home/gmurthy/opensource/qpid-dispatch/src/server.c:1176 #12 0x00007f361cca878e in qd_dispatch_free (qd=0x6164b0) at /home/gmurthy/opensource/qpid-dispatch/src/dispatch.c:318 #13 0x0000000000401864 in main_process (config_path=0x7ffcb36d88e2 "B.conf", python_pkgdir=0x7ffcb36d88ec "/home/gmurthy/opensource/qpid-dispatch/python", fd=2) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:116 #14 0x00000000004022b0 in main (argc=5, argv=0x7ffcb36d8158) at /home/gmurthy/opensource/qpid-dispatch/router/src/main.c:360 (gdb)
Running the test under valgrind, it seems that the pn_proactor_free is trying to free already freed link endpoint. Here are two outputs from valgrind
Process 3493 error: exit code 42, expected 0 qdrouterd -c B.conf -I /home/gmurthy/opensource/qpid-dispatch/python /home/gmurthy/opensource/qpid-dispatch/build/system_test.dir/system_tests_distribution/DistributionTests/setUpClass/B-2.cmd >>>> ==3493== Invalid write of size 8 ==3493== at 0x50E82B0: pn_link_unbound (engine.c:1202) ==3493== by 0x50EB5D0: pni_transport_unbind_handles (transport.c:746) ==3493== by 0x50EB665: pni_transport_unbind_channels (transport.c:761) ==3493== by 0x50EB776: pn_transport_unbind (transport.c:795) ==3493== by 0x50E663D: pn_connection_driver_release_connection (connection_driver.c:81) ==3493== by 0x50E6678: pn_connection_driver_destroy (connection_driver.c:92) ==3493== by 0x530A69B: pconnection_final_free (epoll.c:827) ==3493== by 0x530DB36: pconnection_forced_shutdown (epoll.c:878) ==3493== by 0x530DB36: pn_proactor_free (epoll.c:1815) ==3493== by 0x4EA5BB4: qd_server_free (server.c:1176) ==3493== by 0x4E6678D: qd_dispatch_free (dispatch.c:318) ==3493== by 0x401863: main_process (main.c:116) ==3493== by 0x4022AF: main (main.c:360) ==3493== Address 0x15c06188 is 376 bytes inside a block of size 488 free'd ==3493== at 0x4C2DD18: free (vg_replace_malloc.c:530) ==3493== by 0x50DD938: pn_class_decref (object.c:101) ==3493== by 0x50EA03F: pn_event_finalize (event.c:226) ==3493== by 0x50EA03F: pn_event_finalize_cast (event.c:271) ==3493== by 0x50DD928: pn_class_decref (object.c:95) ==3493== by 0x50EA361: pn_collector_next (event.c:197) ==3493== by 0x50E6408: batch_next (connection_driver.c:51) ==3493== by 0x530C544: pconnection_batch_next (epoll.c:884) ==3493== by 0x4EA50F2: thread_run (server.c:957) ==3493== by 0x551950A: start_thread (in /usr/lib64/libpthread-2.26.so) ==3493== by 0x629916E: clone (in /usr/lib64/libc-2.26.so) ==3493== Block was alloc'd at ==3493== at 0x4C2EA1E: calloc (vg_replace_malloc.c:711) ==3493== by 0x50DD811: pn_object_new (object.c:202) ==3493== by 0x50DD88B: pn_class_new (object.c:61) ==3493== by 0x50E8163: pn_link_new (engine.c:1153) ==3493== by 0x50ED3C4: pn_do_attach (transport.c:1366) ==3493== by 0x50E6227: pni_dispatch_action (dispatcher.c:74) ==3493== by 0x50E6227: pni_dispatch_frame (dispatcher.c:116) ==3493== by 0x50E6227: pn_dispatcher_input (dispatcher.c:135) ==3493== by 0x50EC96B: pn_input_read_amqp (transport.c:2561) ==3493== by 0x50ECA17: transport_consume (transport.c:1817) ==3493== by 0x50F0135: pn_transport_process (transport.c:2946) ==3493== by 0x530C43F: pconnection_process (epoll.c:1128) ==3493== by 0x530C9EA: proactor_do_epoll (epoll.c:2010) ==3493== by 0x4EA50C4: thread_run (server.c:955)
==3488== Invalid read of size 8 ==3488== at 0x50E6E11: pni_ep_get_connection (engine.c:50) ==3488== by 0x50E6E11: pn_ep_decref.part.10 (engine.c:446) ==3488== by 0x50EB587: pni_transport_unbind_handles (transport.c:748) ==3488== by 0x50EB665: pni_transport_unbind_channels (transport.c:761) ==3488== by 0x50EB776: pn_transport_unbind (transport.c:795) ==3488== by 0x50E663D: pn_connection_driver_release_connection (connection_driver.c:81) ==3488== by 0x50E6678: pn_connection_driver_destroy (connection_driver.c:92) ==3488== by 0x530A69B: pconnection_final_free (epoll.c:827) ==3488== by 0x530DF4A: pconnection_done (epoll.c:964) ==3488== by 0x530DF4A: pn_proactor_done (epoll.c:2038) ==3488== by 0x4EA5114: thread_run (server.c:960) ==3488== by 0x551950A: start_thread (in /usr/lib64/libpthread-2.26.so) ==3488== by 0x629916E: clone (in /usr/lib64/libc-2.26.so) ==3488== Address 0x127e15d0 is 400 bytes inside a block of size 488 free'd ==3488== at 0x4C2DD18: free (vg_replace_malloc.c:530) ==3488== by 0x50DD938: pn_class_decref (object.c:101) ==3488== by 0x50EA03F: pn_event_finalize (event.c:226) ==3488== by 0x50EA03F: pn_event_finalize_cast (event.c:271) ==3488== by 0x50DD928: pn_class_decref (object.c:95) ==3488== by 0x50EA361: pn_collector_next (event.c:197) ==3488== by 0x50E6408: batch_next (connection_driver.c:51) ==3488== by 0x530C544: pconnection_batch_next (epoll.c:884) ==3488== by 0x4EA50F2: thread_run (server.c:957) ==3488== by 0x551950A: start_thread (in /usr/lib64/libpthread-2.26.so) ==3488== by 0x629916E: clone (in /usr/lib64/libc-2.26.so) ==3488== Block was alloc'd at ==3488== at 0x4C2EA1E: calloc (vg_replace_malloc.c:711) ==3488== by 0x50DD811: pn_object_new (object.c:202) ==3488== by 0x50DD88B: pn_class_new (object.c:61) ==3488== by 0x50E8163: pn_link_new (engine.c:1153) ==3488== by 0x50ED3C4: pn_do_attach (transport.c:1366) ==3488== by 0x50E6227: pni_dispatch_action (dispatcher.c:74) ==3488== by 0x50E6227: pni_dispatch_frame (dispatcher.c:116) ==3488== by 0x50E6227: pn_dispatcher_input (dispatcher.c:135) ==3488== by 0x50EC96B: pn_input_read_amqp (transport.c:2561) ==3488== by 0x50ECA17: transport_consume (transport.c:1817) ==3488== by 0x50F0135: pn_transport_process (transport.c:2946) ==3488== by 0x530C43F: pconnection_process (epoll.c:1128) ==3488== by 0x530C9EA: proactor_do_epoll (epoll.c:2010) ==3488== by 0x4EA50C4: thread_run (server.c:955)
Attachments
Issue Links
- is related to
-
PROTON-1832 [c] duplicate link names cause invalid read in pn_transport_unbind_handles
-
- Closed
-
- links to
