[PROTON-1842] [c] Dispatch/Proton crashes when opening/closing connections

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: proton-c-0.22.0
• Fix Version/s: proton-c-0.25.0
• Component/s: proton-c
• Assignee: Alan Conway
• Reporter: Charles E. Rolke
• Created: 02/May/18 21:43
• Updated: 06/Sep/18 15:45
• Resolved: 15/Aug/18 18:25

Description

Using proton cpp example code that is modified to open and close connections by the thousands in the main loop and having the event loop short circuit any messaging with:

  void on_connection_open(proton::connection& c) {
      c.close();
{{  }}}

and then directing this client example to a dispatch router 1.1.0. Eventually (after 100,000 to 1,000,000 connection open/closes) the router crashes with:

qdrouterd: /home/chug/git/qpid-proton/c/src/proactor/epoll.c:466: wake_pop_front: Assertion `p->wakes_in_progress' failed.

and with:

qdrouterd: /home/chug/git/qpid-proton/c/src/proactor/epoll.c:2014: proactor_do_epoll: Assertion `ee->type == PCONNECTION_TIMER' failed.

This issue seems to happen only with qpid-dispatch accepting the open/close event stream. Proton cpp example server_direct and c example direct work properly with the same open/close event stream mounting into the 10s of millions of connections.

A core dump backtrace with the PCONNECTION_TIMER failure reads as:

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f795c712c41 in __GI_abort () at abort.c:79
{{#2  0x00007f795c709f7a in __assert_fail_base (fmt=0x7f795c85a260 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7f795d72e15a "ee->type == PCONNECTION_TIMER", }}
{{    file=file@entry=0x7f795d72de98 "/home/chug/git/qpid-proton/c/src/proactor/epoll.c", line=line@entry=2014, }}
    function=function@entry=0x7f795d72e320 <_PRETTY_FUNCTION_.6307> "proactor_do_epoll") at assert.c:92
{{#3  0x00007f795c709ff2 in _GI__assert_fail (assertion=0x7f795d72e15a "ee->type == PCONNECTION_TIMER", file=0x7f795d72de98 "/home/chug/git/qpid-proton/c/src/proactor/epoll.c", line=2014, }}
    function=0x7f795d72e320 <_PRETTY_FUNCTION_.6307> "proactor_do_epoll") at assert.c:101
#4  0x00007f795d72d29f in proactor_do_epoll (p=0x26b7310, can_block=true) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:2014
#5  0x00007f795d72d30e in pn_proactor_wait (p=0x26b7310) at /home/chug/git/qpid-proton/c/src/proactor/epoll.c:2030
#6  0x00007f795dbe89ad in thread_run (arg=0x26be750) at /home/chug/git/qpid-dispatch/src/server.c:946
#7  0x00007f795d50e50b in start_thread (arg=0x7f794f486700) at pthread_create.c:465
#8  0x00007f795c7d216f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

 

Issue Links

• links to: GitHub Pull Request #145