[DISPATCH-385] Router crashes when on-demand host is no longer reachable

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.0
• Fix Version/s: Backlog
• Component/s: Router Node
• Assignee: Ganesh Murthy
• Reporter: Ernest Allen
• Created: 13/Jun/16 16:09
• Updated: 02/Apr/21 11:31

Description

Router will crash if the host for an on-demand connector is unreachable.

To reproduce:
start the 6 router configs in qpid-dispatch/tests/config-6
start a broker on a different box (I used a qpidd on port 11000 on mrg30)
create an on-demand connector to the broker on one of the routers (I use Y)
verify a connection to the broker has been automatically created and is open
kill/block the communications to the broker (I stop my vpn, but there may be better ways)
after a few minutes, the router that the broker was connected to will crash

here is the bt:

#0  0x00007ffff6b9ea28 in raise () from /lib64/libc.so.6
#1  0x00007ffff6ba062a in abort () from /lib64/libc.so.6
#2  0x00007ffff6b97227 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff6b972d2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7b9face in sys_mutex_lock (mutex=0x7fffd4003e00)
    at /home/eallen/workspace/qpid-dispatch/src/posix/threading.c:71
#5  0x00007ffff7bac174 in qdr_forward_deliver_CT (core=0x9264d0, 
    link=0x7fffe403a4c0, dlv=0x7fffe4072ec0)
    at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:132
#6  0x00007ffff7bacfeb in qdr_forward_closest_CT (core=0x9264d0, 
    addr=0x7fffe4024fc0, msg=0x7fffe4060e50, in_delivery=0x0, 
    exclude_inprocess=true, control=false)
    at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:405
#7  0x00007ffff7badd3a in qdr_forward_message_CT (core=0x9264d0, 
    addr=0x7fffe4024fc0, msg=0x7fffe4060e50, in_delivery=0x0, 
    exclude_inprocess=true, control=false)
    at /home/eallen/workspace/qpid-dispatch/src/router_core/forwarder.c:707
#8  0x00007ffff7bb73f6 in qdr_send_to_CT (core=0x9264d0, action=0x9749d0, 
    discard=false)
    at /home/eallen/workspace/qpid-dispatch/src/router_core/transfer.c:581
#9  0x00007ffff7bb18ea in router_core_thread (arg=0x9264d0)
    at /home/eallen/workspace/qpid-dispatch/src/router_core/router_core_thread.c:71
#10 0x00007ffff770d60a in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff6c6c59d in clone () from /lib64/libc.so.6

I'm using master qpid-proton 0.14.0-SNAPSHOT
master qpid-dispatch 0.7.0

Here is the routers debug output after the vpn was killed and just before the crash

Mon Jun 13 11:34:14 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:23 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:34 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:34:44 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.D link removed
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.C link removed
Mon Jun 13 11:35:10 2016 ROUTER (trace) Entered Router Flux Mode
Mon Jun 13 11:35:10 2016 DRIVER (error) getaddrinfo(mrg30.lab.bos.redhat.com, 11000): No address associated with hostname
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.D link set: link_id=0
Mon Jun 13 11:35:10 2016 ROUTER (trace) Node QDR.C link set: link_id=1
qdrouterd: /home/eallen/workspace/qpid-dispatch/src/posix/threading.c:71: sys_mutex_lock: Assertion `result == 0' failed.

Program received signal SIGABRT, Aborted.