[PROTON-2042] Occasional engine.c:691: pni_add_work: Assertion `!delivery->local.settled' failed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: proton-c-0.27.1, proton-c-0.28.0, proton-c-0.29.0, proton-c-0.30.0
• Component/s: python-binding

Environment

To reproduce (I'm on fedora 29)

1) build both qpid-dispatch and qpid-proton with -DCMAKE_BUILD_TYPE=Debug

2) run the delivery abort Qpid dispatch router unit test in a loop:

$ while ctest -VV -R system_tests_delivery_abort; do date; done

The assert usually fires after a few minutes.

• Assignee: Unassigned
• Reporter: Ken Giusti
• Created: 06/May/19 17:54
• Updated: 29/Jan/20 08:44
• Resolved: 29/Jan/20 08:44

Description

When running the Qpid Dispatch Router's unit tests (written in python), occasionally we see the test client (reactor based) core dump on the following (abbreviated) assert:

Program terminated with signal SIGABRT, Aborted.
#0 0x00007fc515270efb in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-26.fc28.x86_64 compat-openssl10-1.0.2o-1.fc28.x86_64 glibc-2.27-38.fc28.x86_64 krb5-libs-1.16.1-25.fc28.x86_64 libdb-5.3.28-30.fc28.x86_64 libuuid-2.32.1-1.fc28.x86_64 libxcrypt-4.4.4-2.fc28.x86_64 pcre2-10.33-1.fc28.x86_64
(gdb) bt
#0 0x00007fc515270efb in raise () from /lib64/libc.so.6
#1 0x00007fc51525b5b9 in abort () from /lib64/libc.so.6
#2 0x00007fc51525b491 in __assert_fail_base.cold.0 () from /lib64/libc.so.6
#3 0x00007fc515269662 in __assert_fail () from /lib64/libc.so.6
#4 0x00007fc5047512d8 in pni_add_work (connection=0x5651b44603e0, delivery=0x5651b44d4920) at /home/kgiusti/work/proton/qpid-proton/c/src/core/engine.c:691
#5 0x00007fc5047514cf in pn_work_update (connection=0x5651b44603e0, delivery=0x5651b44d4920) at /home/kgiusti/work/proton/qpid-proton/c/src/core/engine.c:715
#6 0x00007fc50475437f in pn_link_advance (link=0x5651b4441af0) at /home/kgiusti/work/proton/qpid-proton/c/src/core/engine.c:1796
#7 0x00007fc5049c57ac in _wrap_pn_link_advance (self=0x0, args=(<SwigPyObject at remote 0x7fc501509630>,))
at /home/kgiusti/work/proton/qpid-proton/BUILD/python/CMakeFiles/_cproton.dir/cprotonPYTHON_wrap.c:9539
#8 0x00007fc5161216bb in call_function (oparg=<optimized out>, pp_stack=0x7ffcfa109470) at /usr/src/debug/python2-2.7.15-4.fc28.x86_64/Python/ceval.c:4449
#9 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7fc5014d1578, for file /opt/kgiusti/lib64/proton/bindings/python/proton/_endpoints.py, line 497, in advance (self=<Sender(_impl=<SwigPyObject at remote 0x7fc501509630>, _record=<SwigPyObject at remote 0x7fc501509210>, _attrs=

{'tag_generator': <generator at remote 0x7fc5019e7d20>, 'condition': None}

) at remote 0x7fc5015078d0>), throwflag=throwflag@entry=0) at /usr/src/debug/python2-2.7.15-4.fc28.x86_64/Python/ceval.c:3083
#10 0x00007fc516121e72 in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>,
kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, closure=0x0) at /usr/src/debug/python2-2.7.15-4.fc28.x86_64/Python/ceval.c:3681
#11 0x00007fc51611eaac in fast_function (nk=<optimized out>, na=<optimized out>, n=<optimized out>, pp_stack=0x7ffcfa109630, func=<function at remote 0x7fc501ec5d70>)
at /usr/src/debug/python2-2.7.15-4.fc28.x86_64/Python/ceval.c:4544
#12 call_function (oparg=<optimized out>, pp_stack=0x7ffcfa109630) at /usr/src/debug/python2-2.7.15-4.fc28.x86_64/Python/ceval.c:4469
#13 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7fc501a05400, for file /opt/kgiusti/lib64/proton/bindings/python/proton/_message.py, line 405, in send (self=<Message(body=u'FINISH', _msg=<SwigPyObject at remote 0x7fc501509de0>, annotations=None, _correlation_id=<Data(_data=<SwigPyObject at remote 0x7fc501509c60>, _free=False) at remote 0x7fc5014fa710>, _id=<Data(_data=<SwigPyObject at remote 0x7fc501509b40>, _free=False) at remote 0x7fc5014fab48>, properties=None, instructions=None) at remote 0x7fc5000900d0>, sender=<Sender(_impl=<SwigPyObject at remote 0x7fc501509630>, _record=<SwigPyObject at remote 0x7fc501509210>, _attrs=

{'tag_generator': <generator at remote 0x7fc5019e7d20>, 'condition': None}

) at remote 0x7fc5015078d0>, tag=None, dlv=<Delivery(_impl=<SwigPyObject at remote 0x7fc5015097e0>, _record=<SwigPyObject at remote 0x7fc501509f30>, _attrs={'remote': <Disposition(_impl=<SwigPyObject at remote 0x7fc501509ed0>, _condition=None, _data=None, local=False, _annotations=None) at remote 0x7fc5000902d0>, 'local': <Disposition(_impl=<SwigPyObject ...(truncated), throwflag=throwflag@entry=0) at /usr/src/debug/python2-2.7.15-4.fc28.x86_64/Python/ceval.c:3083

Issue Links

• duplicates: PROTON-2128 Python client crash sending aborted messages
• relates to: PROTON-2005 [proton-c] AMQP error if delivery is aborted with session_outgoing bytes > 0