[DISPATCH-550] Assertion failure in router in larger scale networks

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.7.0
• Component/s: Router Node
• Assignee: Ted Ross
• Reporter: Ted Ross
• Created: 28/Oct/16 12:47
• Updated: 20/Nov/17 18:37
• Resolved: 03/Nov/16 20:00

Description

When running network topologies of six or more router nodes, the following crash is seen intermittently:

#0  0x00007fdf8a448a28 in raise () from /lib64/libc.so.6
#1  0x00007fdf8a44a62a in abort () from /lib64/libc.so.6
#2  0x00007fdf8a441227 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007fdf8a4412d2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fdf8b44dcb1 in sys_mutex_lock (mutex=0x7fdf70032a40) at /home/ross/git/qpid-dispatch/src/posix/threading.c:57
#5  0x00007fdf8b45b442 in qdr_forward_deliver_CT (core=0x123d8b0, link=0x7fdf7004d4a0, dlv=0x7fdf6c03bc20) at /home/ross/git/qpid-dispatch/src/router_core/forwarder.c:154
#6  0x00007fdf8b45c302 in qdr_forward_closest_CT (core=0x123d8b0, addr=0x7fdf78027460, msg=0x7fdf780e5be0, in_delivery=0x0, exclude_inprocess=true, control=true) at /home/ross/git/qpid-dispatch/src/router_core/forwarder.c:436
#7  0x00007fdf8b45d2ad in qdr_forward_message_CT (core=0x123d8b0, addr=0x7fdf78027460, msg=0x7fdf780e5be0, in_delivery=0x0, exclude_inprocess=true, control=true) at /home/ross/git/qpid-dispatch/src/router_core/forwarder.c:749
#8  0x00007fdf8b466b9f in qdr_send_to_CT (core=0x123d8b0, action=0x7fdf6c344d60, discard=false) at /home/ross/git/qpid-dispatch/src/router_core/transfer.c:649
#9  0x00007fdf8b460d91 in router_core_thread (arg=0x123d8b0) at /home/ross/git/qpid-dispatch/src/router_core/router_core_thread.c:83
#10 0x00007fdf8afb761a in start_thread () from /lib64/libpthread.so.0
#11 0x00007fdf8a5165fd in clone () from /lib64/libc.so.6

It appears that the router_core is attempting to deliver a message on a link that is associated with a closed connection. It is also possible that the link itself has been closed and freed.