[DISPATCH-387] Router core dumps with misbehaving client

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.1
• Assignee: Ted Ross
• Reporter: Ulf Lilleengen
• Created: 14/Jun/16 07:33
• Updated: 20/Nov/17 18:39
• Resolved: 14/Jun/16 12:55

Description

I have a simple setup with a router, a sender and a receiver.

The receiver is qpid-receiver from qpid-cpp-client-devel. The sender is a modified version of https://github.com/orpiske/msg-perf-tool, and I've pushed my modifications that made the router crash here: https://github.com/lulf/msg-perf-tool/tree/lulf/dispatch-crash-sender

Using router built from master.

Invoking the receiver: qpid-receive --broker 127.0.0.1:5674 --address amqp-test --ack-frequency 1 --messages 11111 --report-total -f --print-content false
Invoking the sender: ./bin/mpt-sender -b amqp://127.0.0.1:5674/amqp-test -p 1 -d 1 -s 128

The client might not behaving appropriately, so its entirely possible that I don't know what I'm doing! But I was thinking that the router should probably not crash due to misbehaving clients anyway. The core dump can be found here: http://lulf.no/stuff/qdrouterd_dispatch_387.core

(gdb) where
#0 qd_link_pn (link=0x0) at /home/lulf/git/qpid-dispatch/src/container.c:862
#1 0x00007fbf4fc34e0c in CORE_link_push (context=0x1c5e3c0, link=0x1c84b80)
at /home/lulf/git/qpid-dispatch/src/router_node.c:808
#2 0x00007fbf4fc2b27e in qdr_connection_process (conn=0x7fbf380194a0)
at /home/lulf/git/qpid-dispatch/src/router_core/connections.c:175
#3 0x00007fbf4fc19528 in writable_handler (container=0x1b7b510, container=0x1b7b510, conn=<optimized out>,
qd_conn=0x7fbf3800e4f0) at /home/lulf/git/qpid-dispatch/src/container.c:353
#4 handler (handler_context=0x1b7b510, conn_context=<optimized out>, event=<optimized out>, qd_conn=0x7fbf3800e4f0)
at /home/lulf/git/qpid-dispatch/src/container.c:590
#5 0x00007fbf4fc389c5 in process_connector (cxtr=0x7fbf38000950, qd_server=0x1c17ef0)
at /home/lulf/git/qpid-dispatch/src/server.c:744
#6 thread_run (arg=<optimized out>) at /home/lulf/git/qpid-dispatch/src/server.c:964
#7 0x00007fbf4f79961a in start_thread () from /lib64/libpthread.so.0
#8 0x00007fbf4ecf859d in clone () from /lib64/libc.so.6

(gdb) thread apply all bt

Thread 5 (Thread 0x7fbf41fc1700 (LWP 13316)):
#0 0x00007fbf4f79eb20 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf4fc24c8f in sys_cond_wait (cond=<optimized out>, held_mutex=0x1b939c0)
at /home/lulf/git/qpid-dispatch/src/posix/threading.c:107
#2 0x00007fbf4fc38764 in thread_run (arg=<optimized out>) at /home/lulf/git/qpid-dispatch/src/server.c:846
#3 0x00007fbf4f79961a in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbf4ecf859d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fbf50041180 (LWP 13314)):
#0 0x00007fbf4f79eb20 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf4fc24c8f in sys_cond_wait (cond=<optimized out>, held_mutex=0x1b939c0)
at /home/lulf/git/qpid-dispatch/src/posix/threading.c:107
#2 0x00007fbf4fc38764 in thread_run (arg=<optimized out>) at /home/lulf/git/qpid-dispatch/src/server.c:846
#3 0x00007fbf4fc38eb0 in qd_server_run (qd=0x1944030) at /home/lulf/git/qpid-dispatch/src/server.c:1371
#4 0x0000000000401aa7 in main_process (
config_path=config_path@entry=0x7fffe5d8fdf0 "../../qpid-examples/simple_direct.conf",
python_pkgdir=python_pkgdir@entry=0x402420 "../../qpid-install/lib/qpid-dispatch/python", fd=fd@entry=2)
at /home/lulf/git/qpid-dispatch/router/src/main.c:145
#5 0x0000000000401797 in main (argc=3, argv=0x7fffe5d8f348) at /home/lulf/git/qpid-dispatch/router/src/main.c:345

Thread 3 (Thread 0x7fbf429d4700 (LWP 13315)):
#0 0x00007fbf4f79eb20 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fbf4fc24c8f in sys_cond_wait (cond=<optimized out>, held_mutex=0x1c637c0)
at /home/lulf/git/qpid-dispatch/src/posix/threading.c:107
#2 0x00007fbf4fc2ffdd in router_core_thread (arg=0x1c63490)
at /home/lulf/git/qpid-dispatch/src/router_core/router_core_thread.c:54
#3 0x00007fbf4f79961a in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbf4ecf859d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fbf417c0700 (LWP 13317)):
#0 0x00007fbf4ececb1d in poll () from /lib64/libc.so.6
#1 0x00007fbf4fc246e0 in qdpn_driver_wait_2 (d=0x1b2e270, timeout=<optimized out>, timeout@entry=397)
at /home/lulf/git/qpid-dispatch/src/posix/driver.c:964
#2 0x00007fbf4fc38249 in thread_run (arg=<optimized out>) at /home/lulf/git/qpid-dispatch/src/server.c:872
#3 0x00007fbf4f79961a in start_thread () from /lib64/libpthread.so.0
#4 0x00007fbf4ecf859d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fbf40fbf700 (LWP 13318)):
#0 qd_link_pn (link=0x0) at /home/lulf/git/qpid-dispatch/src/container.c:862
#1 0x00007fbf4fc34e0c in CORE_link_push (context=0x1c5e3c0, link=0x1c84b80)
at /home/lulf/git/qpid-dispatch/src/router_node.c:808
#2 0x00007fbf4fc2b27e in qdr_connection_process (conn=0x7fbf380194a0)
at /home/lulf/git/qpid-dispatch/src/router_core/connections.c:175
#3 0x00007fbf4fc19528 in writable_handler (container=0x1b7b510, container=0x1b7b510, conn=<optimized out>,
qd_conn=0x7fbf3800e4f0) at /home/lulf/git/qpid-dispatch/src/container.c:353
#4 handler (handler_context=0x1b7b510, conn_context=<optimized out>, event=<optimized out>, qd_conn=0x7fbf3800e4f0)
at /home/lulf/git/qpid-dispatch/src/container.c:590
#5 0x00007fbf4fc389c5 in process_connector (cxtr=0x7fbf38000950, qd_server=0x1c17ef0)
at /home/lulf/git/qpid-dispatch/src/server.c:744
#6 thread_run (arg=<optimized out>) at /home/lulf/git/qpid-dispatch/src/server.c:964
#7 0x00007fbf4f79961a in start_thread () from /lib64/libpthread.so.0
#8 0x00007fbf4ecf859d in clone () from /lib64/libc.so.6

Issue Links

• is duplicated by: DISPATCH-378 Seg Fault in CORE_link_push
• is duplicated by: DISPATCH-386 Router crashes in qd_link_pn