Uploaded image for project: 'Qpid Dispatch'
  1. Qpid Dispatch
  2. DISPATCH-777

[system_tests_drain] pn_object_free: corrupted double-linked list

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.0.0
    • Tests
    • None

    Description

      Execute the system_tests_drain test suite in Qpid Dispatch until the error appears

      ctest -VV -R system_tests_drain --repeat-until-fail 1000

      There are several very similar errors possible. It either prints

      *** Error in `qdrouterd': corrupted double-linked list: 0x00007f45a40103c0 ***

      or

      ** Error in `qdrouterd': double free or corruption (!prev): 0x00007f49ac013360 ***

      Logs and core files from three failures of the test are attached.

      15: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py" "-m" "unittest" "-v" "system_tests_drain"
      15: Test timeout computed to be: 1500
      *** Error in `qdrouterd': double free or corruption (!prev): 0x00007f49ac013360 ***
      ======= Backtrace: =========
      /lib/x86_64-linux-gnu/libc.so.615: test_drain_support_1_all_messages (system_tests_drain.DrainSupportTest) ... ok
      /lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7f49bc22df96]
      /lib/x86_64-linux-gnu/libc.so.6(+0x7778e)[0x7f49bc22e78e]
      /lib/libqpid-proton-core.so.11(pn_object_free+0x24)[0x7f49bd32a289]
      /lib/libqpid-proton-core.so.11(pn_class_decref+0xca)[0x7f49bd329d8a]
      /lib/libqpid-proton-core.so.11(pn_decref+0x22)[0x7f49bd32a2d2]
      /lib/libqpid-proton-proactor.so.11(+0x42f0)[0x7f49bd1142f0]
      /lib/libqpid-proton-proactor.so.11(+0x4395)[0x7f49bd114395]
      /lib/libqpid-proton-proactor.so.11(+0x4ac5)[0x7f49bd114ac5]
      /lib/libqpid-proton-proactor.so.11(+0x70e5)[0x7f49bd1170e5]
      /lib/libqpid-proton-proactor.so.11(pn_proactor_wait+0x1d)[0x7f49bd117185]
      /main/qpid-dispatch/build/src/libqpid-dispatch.so(+0x55e7a)[0x7f49bd5b8e7a]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7f49bcefa494]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f49bc29f93f]
      ======= Memory map: ========
      55694d2c5000-55694d2c8000 r-xp 00000000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      55694d4c7000-55694d4c8000 r--p 00002000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      55694d4c8000-55694d4c9000 rw-p 00003000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      55694eb26000-55694eea3000 rw-p 00000000 00:00 0                          [heap]
      7f49a4000000-7f49a4021000 rw-p 00000000 00:00 0 
      7f49a4021000-7f49a8000000 ---p 00000000 00:00 0 
      7f49a8000000-7f49a8021000 rw-p 00000000 00:00 0 
      7f49a8021000-7f49ac000000 ---p 00000000 00:00 0 
      7f49ac000000-7f49ac072000 rw-p 00000000 00:00 0 
      7f49ac072000-7f49b0000000 ---p 00000000 00:00 0 
      7f49b21b2000-7f49b21c8000 r-xp 00000000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f49b21c8000-7f49b23c7000 ---p 00016000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f49b23c7000-7f49b23c8000 r--p 00015000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f49b23c8000-7f49b23c9000 rw-p 00016000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f49b23c9000-7f49b23d3000 r-xp 00000000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f49b23d3000-7f49b25d3000 ---p 0000a000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f49b25d3000-7f49b25d4000 r--p 0000a000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f49b25d4000-7f49b25d5000 rw-p 0000b000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f49b25d5000-7f49b25db000 rw-p 00000000 00:00 0 
      7f49b25db000-7f49b25e0000 r-xp 00000000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f49b25e0000-7f49b27df000 ---p 00005000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f49b27df000-7f49b27e0000 r--p 00004000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f49b27e0000-7f49b27e1000 rw-p 00005000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f49b27e1000-7f49b27e5000 r-xp 00000000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f49b27e5000-7f49b29e4000 ---p 00004000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f49b29e4000-7f49b29e5000 r--p 00003000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f49b29e5000-7f49b29e6000 rw-p 00004000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f49b29e6000-7f49b29ee000 r-xp 00000000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f49b29ee000-7f49b2bed000 ---p 00008000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f49b2bed000-7f49b2bee000 r--p 00007000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f49b2bee000-7f49b2bef000 rw-p 00008000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f49b2bef000-7f49b2da3000 r-xp 00000000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f49b2da3000-7f49b2fa3000 ---p 001b4000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f49b2fa3000-7f49b2faa000 r--p 001b4000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f49b2faa000-7f49b2fad000 rw-p 001bb000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f49b2fad000-7f49b2fb3000 r-xp 00000000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f49b2fb3000-7f49b31b2000 ---p 00006000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f49b31b2000-7f49b31b3000 r--p 00005000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f49b31b3000-7f49b31b4000 rw-p 00006000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f49b31b4000-7f49b31bc000 r-xp 00000000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f49b31bc000-7f49b33bc000 ---p 00008000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f49b33bc000-7f49b33bd000 r--p 00008000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f49b33bd000-7f49b33be000 rw-p 00009000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f49b33be000-7f49b33ec000 rw-p 00000000 00:00 0 
      7f49b33ec000-7f49b33f0000 r-xp 00000000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f49b33f0000-7f49b35ef000 ---p 00004000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f49b35ef000-7f49b35f0000 r--p 00003000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f49b35f0000-7f49b35f1000 rw-p 00004000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f49b35f1000-7f49b35fe000 r-xp 00000000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f49b35fe000-7f49b37fd000 ---p 0000d000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f49b37fd000-7f49b37fe000 r--p 0000c000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f49b37fe000-7f49b37ff000 rw-p 0000d000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f49b37ff000-7f49b3800000 ---p 00000000 00:00 0 
      7f49b3800000-7f49b4000000 rw-p 00000000 00:00 0 
      7f49b4000000-7f49b4069000 rw-p 00000000 00:00 0 
      7f49b4069000-7f49b8000000 ---p 00000000 00:00 0 
      7f49b8033000-7f49b8037000 r-xp 00000000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f49b8037000-7f49b8236000 ---p 00004000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f49b8236000-7f49b8237000 r--p 00003000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f49b8237000-7f49b8238000 rw-p 00004000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f49b8238000-7f49b8239000 ---p 00000000 00:00 0 
      7f49b8239000-7f49b8a39000 rw-p 00000000 00:00 0 
      7f49b8a39000-7f49b8a3a000 ---p 00000000 00:00 0 
      7f49b8a3a000-7f49b923a000 rw-p 00000000 00:00 0 
      7f49b923a000-7f49b923b000 ---p 00000000 00:00 0 
      7f49b923b000-7f49b9b3b000 rw-p 00000000 00:00 0 
      7f49b9b3b000-7f49b9d9f000 r-xp 00000000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f49b9d9f000-7f49b9f9f000 ---p 00264000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f49b9f9f000-7f49b9fbd000 r--p 00264000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f49b9fbd000-7f49b9fcb000 rw-p 00282000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f49b9fcb000-7f49b9fce000 rw-p 00000000 00:00 0 
      7f49b9fce000-7f49ba030000 r-xp 00000000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f49ba030000-7f49ba230000 ---p 00062000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f49ba230000-7f49ba234000 r--p 00062000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f49ba234000-7f49ba23a000 rw-p 00066000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f49ba23a000-7f49ba250000 r-xp 00000000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f49ba250000-7f49ba44f000 ---p 00016000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f49ba44f000-7f49ba450000 r--p 00015000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f49ba450000-7f49ba454000 rw-p 00016000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f49ba454000-7f49ba595000 rw-p 00000000 00:00 0 
      7f49ba595000-7f49ba598000 r-xp 00000000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f49ba598000-7f49ba797000 ---p 00003000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f49ba797000-7f49ba798000 r--p 00002000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f49ba798000-7f49ba799000 rw-p 00003000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f49ba799000-7f49ba959000 rw-p 00000000 00:00 0 
      7f49ba959000-7f49ba969000 r-xp 00000000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f49ba969000-7f49bab68000 ---p 00010000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f49bab68000-7f49bab69000 r--p 0000f000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f49bab69000-7f49bab6a000 rw-p 00010000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f49bab6a000-7f49bab71000 r-xp 00000000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f49bab71000-7f49bad71000 ---p 00007000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f49bad71000-7f49bad72000 r--p 00007000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f49bad72000-7f49bad73000 rw-p 00008000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f49bad73000-7f49bad93000 r-xp 00000000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f49bad93000-7f49baf92000 ---p 00020000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f49baf92000-7f49baf93000 r--p 0001f000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f49baf93000-7f49baf97000 rw-p 00020000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f49baf97000-7f49bafab000 r-xp 00000000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f49bafab000-7f49bb1aa000 ---p 00014000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f49bb1aa000-7f49bb1ab000 r--p 00013000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f49bb1ab000-7f49bb1ac000 rw-p 00014000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f49bb1ac000-7f49bb1ae000 rw-p 00000000 00:00 0 
      7f49bb1ae000-7f49bb2b1000 r-xp 00000000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f49bb2b1000-7f49bb4b0000 ---p 00103000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f49bb4b0000-7f49bb4b1000 r--p 00102000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f49bb4b1000-7f49bb4b2000 rw-p 00103000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f49bb4b2000-7f49bb4b4000 r-xp 00000000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f49bb4b4000-7f49bb6b3000 ---p 00002000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f49bb6b3000-7f49bb6b4000 r--p 00001000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f49bb6b4000-7f49bb6b5000 rw-p 00002000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f49bb6b5000-7f49bb6ce000 r-xp 00000000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f49bb6ce000-7f49bb8cd000 ---p 00019000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f49bb8cd000-7f49bb8ce000 r--p 00018000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f49bb8ce000-7f49bb8cf000 rw-p 00019000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f49bb8cf000-7f49bb8e9000 r-xp 00000000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f49bb8e9000-7f49bbae8000 ---p 0001a000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f49bbae8000-7f49bbae9000 r--p 00019000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f49bbae9000-7f49bbaea000 rw-p 0001a000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f49bbaea000-7f49bbd1f000 r-xp 00000000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f49bbd1f000-7f49bbf1f000 ---p 00235000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f49bbf1f000-7f49bbf3b000 r--p 00235000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f49bbf3b000-7f49bbf4a000 rw-p 00251000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f49bbf4a000-7f49bbf4e000 rw-p 00000000 00:00 0 
      7f49bbf4e000-7f49bbfad000 r-xp 00000000 fe:06 402627                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
      7f49bbfad000-7f49bc1ad000 ---p 0005f000 fe:06 402627                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
      7f49bc1ad000-7f49bc1b1000 r--p 0005f000 fe:06 402627                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
      7f49bc1b1000-7f49bc1b7000 rw-p 00063000 fe:06 402627                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
      7f49bc1b7000-7f49bc34c000 r-xp 00000000 fe:06 394121                     /usr/lib/x86_64-linux-gnu/libc-2.24.so
      7f49bc34c000-7f49bc54b000 ---p 00195000 fe:06 394121                     /usr/lib/x86_64-linux-gnu/libc-2.24.so
      7f49bc54b000-7f49bc54f000 r--p 00194000 fe:06 394121                     /usr/lib/x86_64-linux-gnu/libc-2.24.so
      7f49bc54f000-7f49bc551000 rw-p 00198000 fe:06 394121                     /usr/lib/x86_64-linux-gnu/libc-2.24.so
      7f49bc551000-7f49bc555000 rw-p 00000000 00:00 0 
      7f49bc555000-7f49bc84a000 r-xp 00000000 fe:06 402591                     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
      7f49bc84a000-7f49bca4a000 ---p 002f5000 fe:06 402591                     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
      7f49bca4a000-7f49bca4c000 r--p 002f5000 fe:06 402591                     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
      7f49bca4c000-7f49bcac3000 rw-p 002f7000 fe:06 402591                     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
      7f49bcac3000-7f49bcae7000 rw-p 00000000 00:00 0 
      7f49bcae7000-7f49bcaea000 r-xp 00000000 fe:06 394134                     /usr/lib/x86_64-linux-gnu/libdl-2.24.so
      7f49bcaea000-7f49bcce9000 ---p 00003000 fe:06 394134                     /usr/lib/x86_64-linux-gnu/libdl-2.24.so
      7f49bcce9000-7f49bccea000 r--p 00002000 fe:06 394134                     /usr/lib/x86_64-linux-gnu/libdl-2.24.so
      7f49bccea000-7f49bcceb000 rw-p 00003000 fe:06 394134                     /usr/lib/x86_64-linux-gnu/libdl-2.24.so
      7f49bcceb000-7f49bccf2000 r-xp 00000000 fe:06 394207                     /usr/lib/x86_64-linux-gnu/librt-2.24.so
      7f49bccf2000-7f49bcef1000 ---p 00007000 fe:06 394207                     /usr/lib/x86_64-linux-gnu/librt-2.24.so
      7f49bcef1000-7f49bcef2000 r--p 00006000 fe:06 394207                     /usr/lib/x86_64-linux-gnu/librt-2.24.so
      7f49bcef2000-7f49bcef3000 rw-p 00007000 fe:06 394207                     /usr/lib/x86_64-linux-gnu/librt-2.24.so
      7f49bcef3000-7f49bcf0b000 r-xp 00000000 fe:06 394203                     /usr/lib/x86_64-linux-gnu/libpthread-2.24.so
      7f49bcf0b000-7f49bd10a000 ---p 00018000 fe:06 394203                     /usr/lib/x86_64-linux-gnu/libpthread-2.24.so
      7f49bd10a000-7f49bd10b000 r--p 00017000 fe:06 394203                     /usr/lib/x86_64-linux-gnu/libpthread-2.24.so
      7f49bd10b000-7f49bd10c000 rw-p 00018000 fe:06 394203                     /usr/lib/x86_64-linux-gnu/libpthread-2.24.so
      7f49bd10c000-7f49bd110000 rw-p 00000000 00:00 0 
      7f49bd110000-7f49bd11a000 r-xp 00000000 fe:06 412321                     /usr/lib/libqpid-proton-proactor.so.11.0.0
      7f49bd11a000-7f49bd319000 ---p 0000a000 fe:06 412321                     /usr/lib/libqpid-proton-proactor.so.11.0.0
      7f49bd319000-7f49bd31a000 r--p 00009000 fe:06 412321                     /usr/lib/libqpid-proton-proactor.so.11.0.0
      7f49bd31a000-7f49bd31b000 rw-p 0000a000 fe:06 412321                     /usr/lib/libqpid-proton-proactor.so.11.0.0
      7f49bd31b000-7f49bd361000 r-xp 00000000 fe:06 412318                     /usr/lib/libqpid-proton-core.so.11.0.0
      7f49bd361000-7f49bd560000 ---p 00046000 fe:06 412318                     /usr/lib/libqpid-proton-core.so.11.0.0
      7f49bd560000-7f49bd562000 r--p 00045000 fe:06 412318                     /usr/lib/libqpid-proton-core.so.11.0.0
      7f49bd562000-7f49bd563000 rw-p 00047000 fe:06 412318                     /usr/lib/libqpid-proton-core.so.11.0.0
      7f49bd563000-7f49bd5d2000 r-xp 00000000 fe:06 537648                     /main/qpid-dispatch/build/src/libqpid-dispatch.so
      7f49bd5d2000-7f49bd7d2000 ---p 0006f000 fe:06 537648                     /main/qpid-dispatch/build/src/libqpid-dispatch.so
      7f49bd7d2000-7f49bd7d3000 r--p 0006f000 fe:06 537648                     /main/qpid-dispatch/build/src/libqpid-dispatch.so
      7f49bd7d3000-7f49bd7d8000 rw-p 00070000 fe:06 537648                     /main/qpid-dispatch/build/src/libqpid-dispatch.so
      7f49bd7d8000-7f49bd7d9000 rw-p 00000000 00:00 0 
      7f49bd7d9000-7f49bd7fc000 r-xp 00000000 fe:06 394099                     /usr/lib/x86_64-linux-gnu/ld-2.24.so
      7f49bd838000-7f49bd8f8000 rw-p 00000000 00:00 0 
      7f49bd929000-7f49bd9f2000 rw-p 00000000 00:00 0 
      7f49bd9f7000-7f49bd9f8000 rw-p 00000000 00:00 0 
      7f49bd9f8000-7f49bd9f9000 rwxp 00000000 00:00 0 
      7f49bd9f9000-7f49bd9fc000 rw-p 00000000 00:00 0 
      7f49bd9fc000-7f49bd9fd000 r--p 00023000 fe:06 394099                     /usr/lib/x86_64-linux-gnu/ld-2.24.so
      7f49bd9fd000-7f49bd9fe000 rw-p 00024000 fe:06 394099                     /usr/lib/x86_64-linux-gnu/ld-2.24.so
      7f49bd9fe000-7f49bd9ff000 rw-p 00000000 00:00 0 
      7fffc4fe0000-7fffc5001000 rw-p 00000000 00:00 0                          [stack]
      7fffc5005000-7fffc5007000 r--p 00000000 00:00 0                          [vvar]
      7fffc5007000-7fffc5009000 r-xp 00000000 00:00 0                          [vdso]
      ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
      15: test_drain_support_2_one_message (system_tests_drain.DrainSupportTest) ... ERROR:root:proton:io: recv: Connection reset by peer (connection aborted) (connection aborted)
      15: ERROR:root:proton:io: recv: Connection reset by peer (connection aborted) (connection aborted)
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: FAIL
      15: test_drain_support_3_no_messages (system_tests_drain.DrainSupportTest) ... ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: FAIL
      15: test_drain_support_4_no_more_messages (system_tests_drain.DrainSupportTest) ... ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: FAIL
      15: ERROR
      15: 
      15: ======================================================================
      15: ERROR: tearDownClass (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_test.py", line 605, in tearDownClass
      15:     cls.tester.teardown()
      15:   File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
      15:     raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
      15: RuntimeError: Errors during teardown: 
      15: 
      15: Process 1466 error: exit code -6, expected -1
      15: qdrouterd -c test-router.conf -I /main/qpid-dispatch/python
      15: /main/qpid-dispatch/build/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/test-router-1.cmd
      15: >>>>
      15: <<<<
      15: 
      15: ======================================================================
      15: FAIL: test_drain_support_2_one_message (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_tests_drain.py", line 53, in test_drain_support_2_one_message
      15:     self.assertEqual(drain_support.error, None)
      15: AssertionError: 'Timeout Expired: sent: 10 rcvd: 0' != None
      15: 
      15: ======================================================================
      15: FAIL: test_drain_support_3_no_messages (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_tests_drain.py", line 58, in test_drain_support_3_no_messages
      15:     self.assertEqual(drain_support.error, None)
      15: AssertionError: 'Timeout Expired' != None
      15: 
      15: ======================================================================
      15: FAIL: test_drain_support_4_no_more_messages (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_tests_drain.py", line 63, in test_drain_support_4_no_more_messages
      15:     self.assertEqual(drain_support.error, None)
      15: AssertionError: 'Timeout Expired: sent=0 rcvd=0' != None
      15: 
      15: ----------------------------------------------------------------------
      15: Ran 4 tests in 188.313s
      15: 
      15: FAILED (failures=3, errors=1)
          Test #15: system_tests_drain ...............***Failed  188.36 sec
      
      0% tests passed, 1 tests failed out of 1
      
      Total Test time (real) = 209.00 sec
      
      The following tests FAILED:
               15 - system_tests_drain (Failed)
      Errors while running CTest
      
      root@519997eb8015:/main/qpid-dispatch/build# gdb -c /main/qpid-dispatch/build/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/core `which qdrouterd`
      GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
      Copyright (C) 2016 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      Type "show configuration" for configuration details.
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>.
      Find the GDB manual and other documentation resources online at:
      <http://www.gnu.org/software/gdb/documentation/>.
      For help, type "help".
      Type "apropos word" to search for commands related to "word"...
      Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols found)...done.
      [New LWP 1468]
      [New LWP 1467]
      [New LWP 1469]
      [New LWP 1466]
      [New LWP 1470]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `qdrouterd -c test-router.conf -I /main/qpid-dispatch/python'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
      [Current thread is 1 (Thread 0x7f49b9239700 (LWP 1468))]
      (gdb) bt
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      #1  0x00007f49bc1eb3fa in __GI_abort () at abort.c:89
      #2  0x00007f49bc227bd0 in __libc_message (do_abort=do_abort@entry=2, 
          fmt=fmt@entry=0x7f49bc31cbd0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
      #3  0x00007f49bc22df96 in malloc_printerr (action=3, str=0x7f49bc31cc48 "double free or corruption (!prev)", ptr=<optimized out>, 
          ar_ptr=<optimized out>) at malloc.c:5046
      #4  0x00007f49bc22e78e in _int_free (av=0x7f49ac000020, p=0x7f49ac013350, have_lock=0) at malloc.c:3902
      #5  0x00007f49bd32a289 in pn_object_free (object=0x7f49ac013370) at /main/qpid-proton/proton-c/src/core/object/object.c:243
      #6  0x00007f49bd329d8a in pn_class_decref (clazz=0x7f49bd319c20 <pconnection_class>, object=0x7f49ac013370)
          at /main/qpid-proton/proton-c/src/core/object/object.c:101
      #7  0x00007f49bd32a2d2 in pn_decref (object=0x7f49ac013370) at /main/qpid-proton/proton-c/src/core/object/object.c:253
      #8  0x00007f49bd1142f0 in pconnection_final_free (pc=0x7f49ac013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:639
      #9  0x00007f49bd114395 in pconnection_cleanup (pc=0x7f49ac013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:653
      #10 0x00007f49bd114ac5 in pconnection_process (pc=0x7f49ac013370, events=0, timeout=false, topup=false)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:845
      #11 0x00007f49bd1170e5 in proactor_do_epoll (p=0x55694edb1c20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1664
      #12 0x00007f49bd117185 in pn_proactor_wait (p=0x55694edb1c20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #13 0x00007f49bd5b8e7a in thread_run (arg=0x55694eb82370) at /main/qpid-dispatch/src/server.c:817
      #14 0x00007f49bcefa494 in start_thread (arg=0x7f49b9239700) at pthread_create.c:333
      #15 0x00007f49bc29f93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      (gdb) thread apply all bt
      
      Thread 5 (Thread 0x7f49b3fff700 (LWP 1470)):
      #0  0x00007f49bc29ff33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f49bd116fcd in proactor_do_epoll (p=0x55694edb1c20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f49bd117185 in pn_proactor_wait (p=0x55694edb1c20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f49bd5b8e7a in thread_run (arg=0x55694eb82370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f49bcefa494 in start_thread (arg=0x7f49b3fff700) at pthread_create.c:333
      #5  0x00007f49bc29f93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 4 (Thread 0x7f49bd9ea080 (LWP 1466)):
      #0  0x00007f49bc29ff33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f49bd116fcd in proactor_do_epoll (p=0x55694edb1c20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f49bd117185 in pn_proactor_wait (p=0x55694edb1c20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f49bd5b8e7a in thread_run (arg=0x55694eb82370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f49bd5b9987 in qd_server_run (qd=0x55694eb26010) at /main/qpid-dispatch/src/server.c:1031
      #5  0x000055694d2c68d0 in ?? ()
      #6  0x000055694d2c725b in main ()
      
      Thread 3 (Thread 0x7f49b8a38700 (LWP 1469)):
      #0  0x00007f49bc29ff33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f49bd116fcd in proactor_do_epoll (p=0x55694edb1c20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f49bd117185 in pn_proactor_wait (p=0x55694edb1c20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f49bd5b8e7a in thread_run (arg=0x55694eb82370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f49bcefa494 in start_thread (arg=0x7f49b8a38700) at pthread_create.c:333
      #5  0x00007f49bc29f93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 2 (Thread 0x7f49b9a3a700 (LWP 1467)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x00007f49bd5969c0 in sys_cond_wait (cond=0x55694ee285c0, held_mutex=0x55694ee28680)
          at /main/qpid-dispatch/src/posix/threading.c:91
      #2  0x00007f49bd5ac0fa in router_core_thread (arg=0x55694ee282b0) at /main/qpid-dispatch/src/router_core/router_core_thread.c:53
      #3  0x00007f49bcefa494 in start_thread (arg=0x7f49b9a3a700) at pthread_create.c:333
      #4  0x00007f49bc29f93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 1 (Thread 0x7f49b9239700 (LWP 1468)):
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      #1  0x00007f49bc1eb3fa in __GI_abort () at abort.c:89
      #2  0x00007f49bc227bd0 in __libc_message (do_abort=do_abort@entry=2, 
          fmt=fmt@entry=0x7f49bc31cbd0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
      #3  0x00007f49bc22df96 in malloc_printerr (action=3, str=0x7f49bc31cc48 "double free or corruption (!prev)", ptr=<optimized out>, 
          ar_ptr=<optimized out>) at malloc.c:5046
      #4  0x00007f49bc22e78e in _int_free (av=0x7f49ac000020, p=0x7f49ac013350, have_lock=0) at malloc.c:3902
      #5  0x00007f49bd32a289 in pn_object_free (object=0x7f49ac013370) at /main/qpid-proton/proton-c/src/core/object/object.c:243
      #6  0x00007f49bd329d8a in pn_class_decref (clazz=0x7f49bd319c20 <pconnection_class>, object=0x7f49ac013370)
          at /main/qpid-proton/proton-c/src/core/object/object.c:101
      #7  0x00007f49bd32a2d2 in pn_decref (object=0x7f49ac013370) at /main/qpid-proton/proton-c/src/core/object/object.c:253
      #8  0x00007f49bd1142f0 in pconnection_final_free (pc=0x7f49ac013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:639
      #9  0x00007f49bd114395 in pconnection_cleanup (pc=0x7f49ac013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:653
      #10 0x00007f49bd114ac5 in pconnection_process (pc=0x7f49ac013370, events=0, timeout=false, topup=false)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:845
      #11 0x00007f49bd1170e5 in proactor_do_epoll (p=0x55694edb1c20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1664
      #12 0x00007f49bd117185 in pn_proactor_wait (p=0x55694edb1c20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #13 0x00007f49bd5b8e7a in thread_run (arg=0x55694eb82370) at /main/qpid-dispatch/src/server.c:817
      #14 0x00007f49bcefa494 in start_thread (arg=0x7f49b9239700) at pthread_create.c:333
      #15 0x00007f49bc29f93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      (gdb) bt full
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
              set = {__val = {0, 2314913126253082931, 2314885530818453536, 2314885530818453536, 3418922014266630176, 6860733171187738988, 
                  8679965255892022326, 3270858180467255085, 751946305075555890, 7366029038570464823, 3775766004818260068, 
                  3472386802426602041, 3467895053655089200, 2319406791688073264, 4121673501083592038, 2314885531239526452}}
              pid = <optimized out>
              tid = <optimized out>
      #1  0x00007f49bc1eb3fa in __GI_abort () at abort.c:89
              save_stage = 2
              act = {__sigaction_handler = {sa_handler = 0x6666372d30303037, sa_sigaction = 0x6666372d30303037}, sa_mask = {__val = {
                    3472338191832671078, 3467895374536122416, 2319406791620833328, 2319389199435444272, 2314885530818453536, 
                    2314885530818453536, 2314885530818453536, 746875654610770720, 7378697629483820646, 3472328296228087398, 
                    7378697629483820589, 3472329395843524198, 3467895374536122416, 2319406791620833328, 2319389199435444272, 
                    139954615452432}}, sa_flags = 84, sa_restorer = 0x7f49b9238310}
              sigs = {__val = {32, 0 <repeats 15 times>}}
      #2  0x00007f49bc227bd0 in __libc_message (do_abort=do_abort@entry=2, 
          fmt=fmt@entry=0x7f49bc31cbd0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
              ap = {{gp_offset = 40, fp_offset = 1, overflow_arg_area = 0x7f49b9238320, reg_save_area = 0x7f49b92382b0}}
              fd = 9
              on_2 = <optimized out>
              list = <optimized out>
              nlist = <optimized out>
              cp = <optimized out>
              written = <optimized out>
      #3  0x00007f49bc22df96 in malloc_printerr (action=3, str=0x7f49bc31cc48 "double free or corruption (!prev)", ptr=<optimized out>, 
          ar_ptr=<optimized out>) at malloc.c:5046
              buf = "00007f49ac013360"
              cp = <optimized out>
              ar_ptr = <optimized out>
              ptr = <optimized out>
              str = 0x7f49bc31cc48 "double free or corruption (!prev)"
              action = 3
      #4  0x00007f49bc22e78e in _int_free (av=0x7f49ac000020, p=0x7f49ac013350, have_lock=0) at malloc.c:3902
              size = <optimized out>
              fb = <optimized out>
              nextchunk = <optimized out>
              nextsize = <optimized out>
              nextinuse = <optimized out>
              prevsize = <optimized out>
              bck = <optimized out>
              fwd = <optimized out>
              errstr = <optimized out>
              locked = <optimized out>
              __func__ = "_int_free"
      #5  0x00007f49bd32a289 in pn_object_free (object=0x7f49ac013370) at /main/qpid-proton/proton-c/src/core/object/object.c:243
              head = 0x7f49ac013360
      #6  0x00007f49bd329d8a in pn_class_decref (clazz=0x7f49bd319c20 <pconnection_class>, object=0x7f49ac013370)
          at /main/qpid-proton/proton-c/src/core/object/object.c:101
              rc = 0
              __PRETTY_FUNCTION__ = "pn_class_decref"
      #7  0x00007f49bd32a2d2 in pn_decref (object=0x7f49ac013370) at /main/qpid-proton/proton-c/src/core/object/object.c:253
      No locals.
      #8  0x00007f49bd1142f0 in pconnection_final_free (pc=0x7f49ac013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:639
      No locals.
      #9  0x00007f49bd114395 in pconnection_cleanup (pc=0x7f49ac013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:653
              can_free = true
      #10 0x00007f49bd114ac5 in pconnection_process (pc=0x7f49ac013370, events=0, timeout=false, topup=false)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:845
              inbound_wake = false
              timer_unarmed = false
              timer_fired = false
              waking = false
              tick_required = false
              __PRETTY_FUNCTION__ = "pconnection_process"
              closed = false
              unarmed = false
              rearm = false
      #11 0x00007f49bd1170e5 in proactor_do_epoll (p=0x55694edb1c20, can_block=true)
      ---Type <return> to continue, or q <return> to quit---
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1664
              pc = 0x7f49ac013370
              batch = 0x0
              ev = {events = 4, data = {ptr = 0x7f49ac013380, fd = -1409207424, u32 = 2885759872, u64 = 139954395100032}}
              n = 1
              ee = 0x7f49ac013380
              timeout = -1
              __PRETTY_FUNCTION__ = "proactor_do_epoll"
      #12 0x00007f49bd117185 in pn_proactor_wait (p=0x55694edb1c20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      No locals.
      #13 0x00007f49bd5b8e7a in thread_run (arg=0x55694eb82370) at /main/qpid-dispatch/src/server.c:817
              events = 0x7f49ac0138c0
              e = 0x0
              qd_server = 0x55694eb82370
              running = true
      #14 0x00007f49bcefa494 in start_thread (arg=0x7f49b9239700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7f49b9239700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139954615457536, -663125432360584202, 0, 140736498498607, 0, 139954690707520, 
                      603315953528126454, 603309939513146358}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {
                    prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #15 0x00007f49bc29f93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      No locals.
      

      15: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py" "-m" "unittest" "-v" "system_tests_drain"
      15: Test timeout computed to be: 1500
      15: test_drain_support_1_all_messages (system_tests_drain.DrainSupportTest) ... ok
      *** Error in `qdrouterd': corrupted double-linked list: 0x00007f45a40103c0 ***
      ======= Backtrace: =========
      /lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f45bdd38bcb]
      /lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7f45bdd3ef96]
      /lib/x86_64-linux-gnu/libc.so.6(+0x77f6f)[0x7f45bdd3ff6f]
      /lib/libqpid-proton-core.so.11(pn_object_free+0x24)[0x7f45bee3b289]
      /lib/libqpid-proton-core.so.11(pn_class_decref+0xca)[0x7f45bee3ad8a]
      /lib/libqpid-proton-core.so.11(pn_decref+0x22)[0x7f45bee3b2d2]
      /lib/libqpid-proton-proactor.so.11(+0x42f0)[0x7f45bec252f0]
      /lib/libqpid-proton-proactor.so.11(+0x4395)[0x7f45bec25395]
      /lib/libqpid-proton-proactor.so.11(+0x4ac5)[0x7f45bec25ac5]
      /lib/libqpid-proton-proactor.so.11(+0x70e5)[0x7f45bec280e5]
      /lib/libqpid-proton-proactor.so.11(pn_proactor_wait+0x1d)[0x7f45bec28185]
      /main/qpid-dispatch/build/src/libqpid-dispatch.so(+0x55e7a)[0x7f45bf0c9e7a]
      /main/qpid-dispatch/build/src/libqpid-dispatch.so(qd_server_run+0x180)[0x7f45bf0ca987]
      qdrouterd(+0x18d0)[0x5597ff3f88d0]
      qdrouterd(main+0x22d)[0x5597ff3f925b]
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f45bdce82b1]
      qdrouterd(_start+0x2a)[0x5597ff3f85ea]
      ======= Memory map: ========
      15: test_drain_support_2_one_message (system_tests_drain.DrainSupportTest) ... ok
      5597ff3f7000-5597ff3fa000 r-xp 00000000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      5597ff5f9000-5597ff5fa000 r--p 00002000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      5597ff5fa000-5597ff5fb000 rw-p 00003000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      559800530000-55980088c000 rw-p 00000000 00:00 0                          [heap]
      7f45a4000000-7f45a404e000 rw-p 00000000 00:00 0 
      7f45a404e000-7f45a8000000 ---p 00000000 00:00 0 
      7f45abbd7000-7f45abbed000 r-xp 00000000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f45abbed000-7f45abdec000 ---p 00016000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f45abdec000-7f45abded000 r--p 00015000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f45abded000-7f45abdee000 rw-p 00016000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f45abdee000-7f45abdf8000 r-xp 00000000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f45abdf8000-7f45abff8000 ---p 0000a000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f45abff8000-7f45abff9000 r--p 0000a000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f45abff9000-7f45abffa000 rw-p 0000b000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f45abffa000-7f45ac000000 rw-p 00000000 00:00 0 
      7f45ac000000-7f45ac023000 rw-p 00000000 00:00 0 
      7f45ac023000-7f45b0000000 ---p 00000000 00:00 0 
      7f45b0000000-7f45b0075000 rw-p 00000000 00:00 0 
      7f45b0075000-7f45b4000000 ---p 00000000 00:00 0 
      7f45b4000000-7f45b407a000 rw-p 00000000 00:00 0 
      7f45b407a000-7f45b8000000 ---p 00000000 00:00 0 
      7f45b811f000-7f45b8124000 r-xp 00000000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f45b8124000-7f45b8323000 ---p 00005000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f45b8323000-7f45b8324000 r--p 00004000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f45b8324000-7f45b8325000 rw-p 00005000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f45b8325000-7f45b8329000 r-xp 00000000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f45b8329000-7f45b8528000 ---p 00004000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f45b8528000-7f45b8529000 r--p 00003000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f45b8529000-7f45b852a000 rw-p 00004000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f45b852a000-7f45b8532000 r-xp 00000000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f45b8532000-7f45b8731000 ---p 00008000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f45b8731000-7f45b8732000 r--p 00007000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f45b8732000-7f45b8733000 rw-p 00008000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f45b8733000-7f45b88e7000 r-xp 00000000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f45b88e7000-7f45b8ae7000 ---p 001b4000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f45b8ae7000-7f45b8aee000 r--p 001b4000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f45b8aee000-7f45b8af1000 rw-p 001bb000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f45b8af1000-7f45b8af7000 r-xp 00000000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f45b8af7000-7f45b8cf6000 ---p 00006000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f45b8cf6000-7f45b8cf7000 r--p 00005000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f45b8cf7000-7f45b8cf8000 rw-p 00006000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f45b8cf8000-7f45b8d00000 r-xp 00000000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f45b8d00000-7f45b8f00000 ---p 00008000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f45b8f00000-7f45b8f01000 r--p 00008000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f45b8f01000-7f45b8f02000 rw-p 00009000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f45b8f02000-7f45b8f30000 rw-p 00000000 00:00 0 
      7f45b8f30000-7f45b8f34000 r-xp 00000000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f45b8f34000-7f45b9133000 ---p 00004000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f45b9133000-7f45b9134000 r--p 00003000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f45b9134000-7f45b9135000 rw-p 00004000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f45b9135000-7f45b9142000 r-xp 00000000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f45b9142000-7f45b9341000 ---p 0000d000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f45b9341000-7f45b9342000 r--p 0000c000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f45b9342000-7f45b9343000 rw-p 0000d000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f45b9343000-7f45b9347000 r-xp 00000000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f45b9347000-7f45b9546000 ---p 00004000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f45b9546000-7f45b9547000 r--p 00003000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f45b9547000-7f45b9548000 rw-p 00004000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f45b9548000-7f45b9549000 ---p 00000000 00:00 0 
      7f45b9549000-7f45b9d49000 rw-p 00000000 00:00 0 
      7f45b9d49000-7f45b9d4a000 ---p 00000000 00:00 0 
      7f45b9d4a000-7f45ba54a000 rw-p 00000000 00:00 0 
      7f45ba54a000-7f45ba54b000 ---p 00000000 00:00 0 
      7f45ba54b000-7f45bad4b000 rw-p 00000000 00:00 0 
      7f45bad4b000-7f45bad4c000 ---p 00000000 00:00 0 
      7f45bad4c000-7f45bb64c000 rw-p 00000000 00:00 0 
      7f45bb64c000-7f45bb8b0000 r-xp 00000000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f45bb8b0000-7f45bbab0000 ---p 00264000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f45bbab0000-7f45bbace000 r--p 00264000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f45bbace000-7f45bbadc000 rw-p 00282000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f45bbadc000-7f45bbadf000 rw-p 00000000 00:00 0 
      7f45bbadf000-7f45bbb41000 r-xp 00000000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f45bbb41000-7f45bbd41000 ---p 00062000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f45bbd41000-7f45bbd45000 r--p 00062000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f45bbd45000-7f45bbd4b000 rw-p 00066000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f45bbd4b000-7f45bbd61000 r-xp 00000000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f45bbd61000-7f45bbf60000 ---p 00016000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f45bbf60000-7f45bbf61000 r--p 00015000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f45bbf61000-7f45bbf65000 rw-p 00016000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f45bbf65000-7f45bc0a6000 rw-p 00000000 00:00 0 
      7f45bc0a6000-7f45bc0a9000 r-xp 00000000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f45bc0a9000-7f45bc2a8000 ---p 00003000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f45bc2a8000-7f45bc2a9000 r--p 00002000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f45bc2a9000-7f45bc2aa000 rw-p 00003000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f45bc2aa000-7f45bc46a000 rw-p 00000000 00:00 0 
      7f45bc46a000-7f45bc47a000 r-xp 00000000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f45bc47a000-7f45bc679000 ---p 00010000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f45bc679000-7f45bc67a000 r--p 0000f000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f45bc67a000-7f45bc67b000 rw-p 00010000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f45bc67b000-7f45bc682000 r-xp 00000000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f45bc682000-7f45bc882000 ---p 00007000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f45bc882000-7f45bc883000 r--p 00007000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f45bc883000-7f45bc884000 rw-p 00008000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f45bc884000-7f45bc8a4000 r-xp 00000000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f45bc8a4000-7f45bcaa3000 ---p 00020000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f45bcaa3000-7f45bcaa4000 r--p 0001f000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f45bcaa4000-7f45bcaa8000 rw-p 00020000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f45bcaa8000-7f45bcabc000 r-xp 00000000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f45bcabc000-7f45bccbb000 ---p 00014000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f45bccbb000-7f45bccbc000 r--p 00013000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f45bccbc000-7f45bccbd000 rw-p 00014000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f45bccbd000-7f45bccbf000 rw-p 00000000 00:00 0 
      7f45bccbf000-7f45bcdc2000 r-xp 00000000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f45bcdc2000-7f45bcfc1000 ---p 00103000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f45bcfc1000-7f45bcfc2000 r--p 00102000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f45bcfc2000-7f45bcfc3000 rw-p 00103000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f45bcfc3000-7f45bcfc5000 r-xp 00000000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f45bcfc5000-7f45bd1c4000 ---p 00002000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f45bd1c4000-7f45bd1c5000 r--p 00001000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f45bd1c5000-7f45bd1c6000 rw-p 00002000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f45bd1c6000-7f45bd1df000 r-xp 00000000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f45bd1df000-7f45bd3de000 ---p 00019000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f45bd3de000-7f45bd3df000 r--p 00018000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f45bd3df000-7f45bd3e0000 rw-p 00019000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f45bd3e0000-7f45bd3fa000 r-xp 00000000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f45bd3fa000-7f45bd5f9000 ---p 0001a000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f45bd5f9000-7f45bd5fa000 r--p 00019000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f45bd5fa000-7f45bd5fb000 rw-p 0001a000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f45bd5fb000-7f45bd830000 r-xp 00000000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f45bd830000-7f45bda30000 ---p 00235000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f45bda30000-7f45bda4c000 r--p 00235000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f45bda4c000-7f45bda5b000 rw-p 00251000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f45bda5b000-7f45bda5f000 rw-p 00000000 00:00 0 15: test_drain_support_3_no_messages (system_tests_drain.DrainSupportTest) ... ERROR:root:proton:io: recv: Connection reset by peer
      15: ERROR:root:proton:io: recv: Connection reset by peer
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: FAIL
      15: test_drain_support_4_no_more_messages (system_tests_drain.DrainSupportTest) ... ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: FAIL
      15: ERROR
      15: 
      15: ======================================================================
      15: ERROR: tearDownClass (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_test.py", line 605, in tearDownClass
      15:     cls.tester.teardown()
      15:   File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
      15:     raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
      15: RuntimeError: Errors during teardown: 
      15: 
      15: Process 1920 error: exit code -6, expected -1
      15: qdrouterd -c test-router.conf -I /main/qpid-dispatch/python
      15: /main/qpid-dispatch/build/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/test-router-1.cmd
      15: >>>>
      15: <<<<
      15: 
      15: ======================================================================
      15: FAIL: test_drain_support_3_no_messages (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_tests_drain.py", line 58, in test_drain_support_3_no_messages
      15:     self.assertEqual(drain_support.error, None)
      15: AssertionError: 'Timeout Expired' != None
      15: 
      15: ======================================================================
      15: FAIL: test_drain_support_4_no_more_messages (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_tests_drain.py", line 63, in test_drain_support_4_no_more_messages
      15:     self.assertEqual(drain_support.error, None)
      15: AssertionError: 'Timeout Expired: sent=0 rcvd=0' != None
      15: 
      15: ----------------------------------------------------------------------
      15: Ran 4 tests in 125.603s
      15: 
      15: FAILED (failures=2, errors=1)
          Test #15: system_tests_drain ...............***Failed  125.66 sec
      
      0% tests passed, 1 tests failed out of 1
      
      Total Test time (real) = 142.44 sec
      
      The following tests FAILED:
               15 - system_tests_drain (Failed)
      Errors while running CTest
      
      root@519997eb8015:/main/qpid-dispatch/build# find / -name core
      /main/qpid-dispatch/build/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/core
      /main/qpid-proton/build/proton-c/CMakeFiles/qpid-proton-core.dir/src/core
      /main/qpid-proton/build/proton-c/CMakeFiles/qpid-proton.dir/src/core
      /main/qpid-proton/build/proton-c/bindings/python/dist/proton-c/src/core
      /main/qpid-proton/proton-c/src/core
      /main/qpid-proton/proton-c/bindings/ruby/lib/core
      /proc/sys/net/core
      /dev/core
      root@519997eb8015:/main/qpid-dispatch/build# gdb -c /main/qpid-dispatch/build/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/core `which qdrouterd`
      GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
      Copyright (C) 2016 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      Type "show configuration" for configuration details.
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>.
      Find the GDB manual and other documentation resources online at:
      <http://www.gnu.org/software/gdb/documentation/>.
      For help, type "help".
      Type "apropos word" to search for commands related to "word"...
      Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols found)...done.
      [New LWP 1920]
      [New LWP 1924]
      [New LWP 1921]
      [New LWP 1923]
      [New LWP 1922]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `qdrouterd -c test-router.conf -I /main/qpid-dispatch/python'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
      [Current thread is 1 (Thread 0x7f45bf4fb080 (LWP 1920))]
      (gdb) bt full
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
              set = {__val = {0, 4122257324577660982, 2314885530818453536, 2314885530818453536, 8319607701661294624, 4069054363051241330, 
                  7956009158131998518, 7795578597039503477, 8031167702420906601, 3328210913394127662, 7233402450679892530, 
                  3975887029563044705, 7148445592316097638, 8083166874389458992, 3472333806704275488, 2321095685323497520}}
              pid = <optimized out>
              tid = <optimized out>
      #1  0x00007f45bdcfc3fa in __GI_abort () at abort.c:89
              save_stage = 2
              act = {__sigaction_handler = {sa_handler = 0x6620303030303030, sa_sigaction = 0x6620303030303030}, sa_mask = {__val = {
                    3616447813731236453, 2314885530819965494, 2314885530818453536, 2314885530818453536, 7091318039310988591, 
                    3257288213055174703, 7955377262162766188, 7814716644930629493, 3328210913394127662, 7233402450679892530, 
                    3975887029566530145, 7305510828343899238, 8083166578036715568, 3472387691330416672, 2321095685323497520, 
                    140724608901328}}, sa_flags = 79, sa_restorer = 0x7ffd005320d0}
              sigs = {__val = {32, 0 <repeats 15 times>}}
      #2  0x00007f45bdd38bd0 in __libc_message (do_abort=do_abort@entry=2, 
          fmt=fmt@entry=0x7f45bde2dbd0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
              ap = {{gp_offset = 40, fp_offset = 32581, overflow_arg_area = 0x7ffd005320e0, reg_save_area = 0x7ffd00532070}}
              fd = 9
              on_2 = <optimized out>
              list = <optimized out>
              nlist = <optimized out>
              cp = <optimized out>
              written = <optimized out>
      #3  0x00007f45bdd3ef96 in malloc_printerr (action=3, str=0x7f45bde2a71e "corrupted double-linked list", ptr=<optimized out>, 
          ar_ptr=<optimized out>) at malloc.c:5046
              buf = "00007f45a40103c0"
              cp = <optimized out>
              ar_ptr = <optimized out>
              ptr = <optimized out>
              str = 0x7f45bde2a71e "corrupted double-linked list"
              action = 3
      #4  0x00007f45bdd3ff6f in _int_free (av=0x7f45a4000020, p=0x7f45a400f9a0, have_lock=0) at malloc.c:4049
              size = 2592
              fb = <optimized out>
              nextchunk = 0x7f45a40103c0
              nextsize = 960
              nextinuse = <optimized out>
              prevsize = <optimized out>
              bck = <optimized out>
              fwd = <optimized out>
              errstr = 0x0
              locked = <optimized out>
              __func__ = "_int_free"
      #5  0x00007f45bee3b289 in pn_object_free (object=0x7f45a400fd40) at /main/qpid-proton/proton-c/src/core/object/object.c:243
              head = 0x7f45a400fd30
      #6  0x00007f45bee3ad8a in pn_class_decref (clazz=0x7f45bee2ac20 <pconnection_class>, object=0x7f45a400fd40)
          at /main/qpid-proton/proton-c/src/core/object/object.c:101
              rc = 0
              __PRETTY_FUNCTION__ = "pn_class_decref"
      #7  0x00007f45bee3b2d2 in pn_decref (object=0x7f45a400fd40) at /main/qpid-proton/proton-c/src/core/object/object.c:253
      No locals.
      #8  0x00007f45bec252f0 in pconnection_final_free (pc=0x7f45a400fd40) at /main/qpid-proton/proton-c/src/proactor/epoll.c:639
      No locals.
      #9  0x00007f45bec25395 in pconnection_cleanup (pc=0x7f45a400fd40) at /main/qpid-proton/proton-c/src/proactor/epoll.c:653
              can_free = true
      #10 0x00007f45bec25ac5 in pconnection_process (pc=0x7f45a400fd40, events=0, timeout=false, topup=false)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:845
              inbound_wake = false
              timer_unarmed = false
              timer_fired = false
              waking = false
              tick_required = false
              __PRETTY_FUNCTION__ = "pconnection_process"
              closed = false
              unarmed = false
              rearm = false
      #11 0x00007f45bec280e5 in proactor_do_epoll (p=0x5598007bbc20, can_block=true)
      ---Type <return> to continue, or q <return> to quit---
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1664
              pc = 0x7f45a400fd40
              batch = 0x0
              ev = {events = 4, data = {ptr = 0x7f45a400fd50, fd = -1543439024, u32 = 2751528272, u64 = 139937080999248}}
              n = 1
              ee = 0x7f45a400fd50
              timeout = -1
              __PRETTY_FUNCTION__ = "proactor_do_epoll"
      #12 0x00007f45bec28185 in pn_proactor_wait (p=0x5598007bbc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      No locals.
      #13 0x00007f45bf0c9e7a in thread_run (arg=0x55980058c370) at /main/qpid-dispatch/src/server.c:817
              events = 0x7f45a4010290
              e = 0x0
              qd_server = 0x55980058c370
              running = true
      #14 0x00007f45bf0ca987 in qd_server_run (qd=0x559800530010) at /main/qpid-dispatch/src/server.c:1031
              qd_server = 0x55980058c370
              i = 3
              __PRETTY_FUNCTION__ = "qd_server_run"
              n = 3
              threads = 0x5598008256a0
      #15 0x00005597ff3f88d0 in ?? ()
      No symbol table info available.
      #16 0x00005597ff3f925b in main ()
      No symbol table info available.
      (gdb) 
      
      
      (gdb) 
      (gdb) thread apply all bt
      
      Thread 5 (Thread 0x7f45bad4a700 (LWP 1922)):
      #0  0x00007f45bddb0f33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f45bec27fcd in proactor_do_epoll (p=0x5598007bbc20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f45bec28185 in pn_proactor_wait (p=0x5598007bbc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f45bf0c9e7a in thread_run (arg=0x55980058c370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f45bea0b494 in start_thread (arg=0x7f45bad4a700) at pthread_create.c:333
      #5  0x00007f45bddb093f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 4 (Thread 0x7f45ba549700 (LWP 1923)):
      #0  0x00007f45bddb0f33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f45bec27fcd in proactor_do_epoll (p=0x5598007bbc20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f45bec28185 in pn_proactor_wait (p=0x5598007bbc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f45bf0c9e7a in thread_run (arg=0x55980058c370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f45bea0b494 in start_thread (arg=0x7f45ba549700) at pthread_create.c:333
      #5  0x00007f45bddb093f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 3 (Thread 0x7f45bb54b700 (LWP 1921)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x00007f45bf0a79c0 in sys_cond_wait (cond=0x5598008325c0, held_mutex=0x559800832680)
          at /main/qpid-dispatch/src/posix/threading.c:91
      #2  0x00007f45bf0bd0fa in router_core_thread (arg=0x5598008322b0) at /main/qpid-dispatch/src/router_core/router_core_thread.c:53
      #3  0x00007f45bea0b494 in start_thread (arg=0x7f45bb54b700) at pthread_create.c:333
      #4  0x00007f45bddb093f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 2 (Thread 0x7f45b9d48700 (LWP 1924)):
      #0  0x00007f45bddb0f33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f45bec27fcd in proactor_do_epoll (p=0x5598007bbc20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f45bec28185 in pn_proactor_wait (p=0x5598007bbc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f45bf0c9e7a in thread_run (arg=0x55980058c370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f45bea0b494 in start_thread (arg=0x7f45b9d48700) at pthread_create.c:333
      #5  0x00007f45bddb093f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 1 (Thread 0x7f45bf4fb080 (LWP 1920)):
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      #1  0x00007f45bdcfc3fa in __GI_abort () at abort.c:89
      #2  0x00007f45bdd38bd0 in __libc_message (do_abort=do_abort@entry=2, 
          fmt=fmt@entry=0x7f45bde2dbd0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
      #3  0x00007f45bdd3ef96 in malloc_printerr (action=3, str=0x7f45bde2a71e "corrupted double-linked list", ptr=<optimized out>, 
          ar_ptr=<optimized out>) at malloc.c:5046
      #4  0x00007f45bdd3ff6f in _int_free (av=0x7f45a4000020, p=0x7f45a400f9a0, have_lock=0) at malloc.c:4049
      #5  0x00007f45bee3b289 in pn_object_free (object=0x7f45a400fd40) at /main/qpid-proton/proton-c/src/core/object/object.c:243
      #6  0x00007f45bee3ad8a in pn_class_decref (clazz=0x7f45bee2ac20 <pconnection_class>, object=0x7f45a400fd40)
          at /main/qpid-proton/proton-c/src/core/object/object.c:101
      #7  0x00007f45bee3b2d2 in pn_decref (object=0x7f45a400fd40) at /main/qpid-proton/proton-c/src/core/object/object.c:253
      #8  0x00007f45bec252f0 in pconnection_final_free (pc=0x7f45a400fd40) at /main/qpid-proton/proton-c/src/proactor/epoll.c:639
      #9  0x00007f45bec25395 in pconnection_cleanup (pc=0x7f45a400fd40) at /main/qpid-proton/proton-c/src/proactor/epoll.c:653
      #10 0x00007f45bec25ac5 in pconnection_process (pc=0x7f45a400fd40, events=0, timeout=false, topup=false)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:845
      #11 0x00007f45bec280e5 in proactor_do_epoll (p=0x5598007bbc20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1664
      #12 0x00007f45bec28185 in pn_proactor_wait (p=0x5598007bbc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #13 0x00007f45bf0c9e7a in thread_run (arg=0x55980058c370) at /main/qpid-dispatch/src/server.c:817
      #14 0x00007f45bf0ca987 in qd_server_run (qd=0x559800530010) at /main/qpid-dispatch/src/server.c:1031
      #15 0x00005597ff3f88d0 in ?? ()
      #16 0x00005597ff3f925b in main ()
      

      15: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py" "-m" "unittest" "-v" "system_tests_drain"
      15: Test timeout computed to be: 1500
      *** Error in `qdrouterd': double free or corruption (!prev): 0x00007f6d80013360 ***
      ======= Backtrace: =========
      /lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f6d92155bcb]
      /lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7f6d9215bf96]
      /lib/x86_64-linux-gnu/libc.so.6(+0x7778e)[0x7f6d9215c78e]
      /lib/libqpid-proton-core.so.11(pn_object_free+0x24)[0x7f6d93258289]
      /lib/libqpid-proton-core.so.11(pn_class_decref+0xca)[0x7f6d93257d8a]
      /lib/libqpid-proton-core.so.11(pn_decref+0x22)[0x7f6d932582d2]
      /lib/libqpid-proton-proactor.so.11(+0x42f0)[0x7f6d930422f0]
      /lib/libqpid-proton-proactor.so.11(+0x4395)[0x7f6d93042395]
      /lib/libqpid-proton-proactor.so.11(+0x4ac5)[0x7f6d93042ac5]
      /lib/libqpid-proton-proactor.so.11(+0x70e5)[0x7f6d930450e5]
      /lib/libqpid-proton-proactor.so.11(pn_proactor_wait+0x1d)[0x7f6d93045185]
      /main/qpid-dispatch/build/src/libqpid-dispatch.so(+0x55e7a)[0x7f6d934e6e7a]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7f6d92e28494]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f6d921cd93f]
      ======= Memory map: ========
      557e3390a000-557e3390d000 r-xp 00000000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      557e33b0c000-557e33b0d000 r--p 00002000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      557e33b0d000-557e33b0e000 rw-p 00003000 fe:06 537578                     /main/qpid-dispatch/build/router/qdrouterd
      557e34702000-557e34a5e000 rw-p 00000000 00:00 0                          [heap]
      7f6d78000000-7f6d78021000 rw-p 00000000 00:00 0 
      7f6d78021000-7f6d7c000000 ---p 00000000 00:00 0 
      7f6d7c000000-7f6d7c021000 rw-p 00000000 00:00 0 
      7f6d7c021000-7f6d80000000 ---p 00000000 00:00 0 
      7f6d80000000-7f6d80072000 rw-p 00000000 00:00 0 
      7f6d80072000-7f6d84000000 ---p 00000000 00:00 0 
      7f6d84000000-7f6d84044000 rw-p 00000000 00:00 0 
      7f6d84044000-7f6d88000000 ---p 00000000 00:00 0 
      7f6d88000000-7f6d88069000 rw-p 00000000 00:00 0 
      7f6d88069000-7f6d8c000000 ---p 00000000 00:00 0 
      7f6d8c113000-7f6d8c129000 r-xp 00000000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f6d8c129000-7f6d8c328000 ---p 00016000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f6d8c328000-7f6d8c329000 r--p 00015000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f6d8c329000-7f6d8c32a000 rw-p 00016000 fe:06 394148                     /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
      7f6d8c32a000-7f6d8c334000 r-xp 00000000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f6d8c334000-7f6d8c534000 ---p 0000a000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f6d8c534000-7f6d8c535000 r--p 0000a000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f6d8c535000-7f6d8c536000 rw-p 0000b000 fe:06 394180                     /usr/lib/x86_64-linux-gnu/libnss_files-2.24.so
      7f6d8c536000-7f6d8c53c000 rw-p 00000000 00:00 0 
      7f6d8c53c000-7f6d8c541000 r-xp 00000000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f6d8c541000-7f6d8c740000 ---p 00005000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f6d8c740000-7f6d8c741000 r--p 00004000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f6d8c741000-7f6d8c742000 rw-p 00005000 fe:06 530154                     /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25
      7f6d8c742000-7f6d8c746000 r-xp 00000000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f6d8c746000-7f6d8c945000 ---p 00004000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f6d8c945000-7f6d8c946000 r--p 00003000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f6d8c946000-7f6d8c947000 rw-p 00004000 fe:06 530160                     /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25
      7f6d8c947000-7f6d8c94f000 r-xp 00000000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f6d8c94f000-7f6d8cb4e000 ---p 00008000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f6d8cb4e000-7f6d8cb4f000 r--p 00007000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f6d8cb4f000-7f6d8cb50000 rw-p 00008000 fe:06 530163                     /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25
      7f6d8cb50000-7f6d8cd04000 r-xp 00000000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f6d8cd04000-7f6d8cf04000 ---p 001b4000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f6d8cf04000-7f6d8cf0b000 r--p 001b4000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f6d8cf0b000-7f6d8cf0e000 rw-p 001bb000 fe:06 394131                     /usr/lib/x86_64-linux-gnu/libdb-5.3.so
      7f6d8cf0e000-7f6d8cf14000 r-xp 00000000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f6d8cf14000-7f6d8d113000 ---p 00006000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f6d8d113000-7f6d8d114000 r--p 00005000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f6d8d114000-7f6d8d115000 rw-p 00006000 fe:06 530169                     /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25
      7f6d8d115000-7f6d8d11d000 r-xp 00000000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f6d8d11d000-7f6d8d31d000 ---p 00008000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f6d8d31d000-7f6d8d31e000 r--p 00008000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f6d8d31e000-7f6d8d31f000 rw-p 00009000 fe:06 394129                     /usr/lib/x86_64-linux-gnu/libcrypt-2.24.so
      7f6d8d31f000-7f6d8d34d000 rw-p 00000000 00:00 0 
      7f6d8d34d000-7f6d8d351000 r-xp 00000000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f6d8d351000-7f6d8d550000 ---p 00004000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f6d8d550000-7f6d8d551000 r--p 00003000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f6d8d551000-7f6d8d552000 rw-p 00004000 fe:06 530166                     /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25
      7f6d8d552000-7f6d8d55f000 r-xp 00000000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f6d8d55f000-7f6d8d75e000 ---p 0000d000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f6d8d75e000-7f6d8d75f000 r--p 0000c000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f6d8d75f000-7f6d8d760000 rw-p 0000d000 fe:06 530157                     /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25
      7f6d8d760000-7f6d8d764000 r-xp 00000000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f6d8d764000-7f6d8d963000 ---p 00004000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f6d8d963000-7f6d8d964000 r--p 00003000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f6d8d964000-7f6d8d965000 rw-p 00004000 fe:06 530151                     /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25
      7f6d8d965000-7f6d8d966000 ---p 00000000 00:00 0 
      7f6d8d966000-7f6d8e166000 rw-p 00000000 00:00 0 
      7f6d8e166000-7f6d8e167000 ---p 00000000 00:00 0 
      7f6d8e167000-7f6d8e967000 rw-p 00000000 00:00 0 
      7f6d8e967000-7f6d8e968000 ---p 00000000 00:00 0 
      7f6d8e968000-7f6d8f168000 rw-p 00000000 00:00 0 
      7f6d8f168000-7f6d8f169000 ---p 00000000 00:00 0 
      7f6d8f169000-7f6d8fa69000 rw-p 00000000 00:00 0 
      7f6d8fa69000-7f6d8fccd000 r-xp 00000000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f6d8fccd000-7f6d8fecd000 ---p 00264000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f6d8fecd000-7f6d8feeb000 r--p 00264000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f6d8feeb000-7f6d8fef9000 rw-p 00282000 fe:06 402360                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
      7f6d8fef9000-7f6d8fefc000 rw-p 00000000 00:00 0 
      7f6d8fefc000-7f6d8ff5e000 r-xp 00000000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f6d8ff5e000-7f6d9015e000 ---p 00062000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f6d9015e000-7f6d90162000 r--p 00062000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f6d90162000-7f6d90168000 rw-p 00066000 fe:06 402628                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
      7f6d90168000-7f6d9017e000 r-xp 00000000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f6d9017e000-7f6d9037d000 ---p 00016000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f6d9037d000-7f6d9037e000 r--p 00015000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f6d9037e000-7f6d90382000 rw-p 00016000 fe:06 400927                     /usr/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so
      7f6d90382000-7f6d904c3000 rw-p 00000000 00:00 0 
      7f6d904c3000-7f6d904c6000 r-xp 00000000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f6d904c6000-7f6d906c5000 ---p 00003000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f6d906c5000-7f6d906c6000 r--p 00002000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f6d906c6000-7f6d906c7000 rw-p 00003000 fe:06 400923                     /usr/lib/python2.7/lib-dynload/_lsprof.x86_64-linux-gnu.so
      7f6d906c7000-7f6d90887000 rw-p 00000000 00:00 0 
      7f6d90887000-7f6d90897000 r-xp 00000000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f6d90897000-7f6d90a96000 ---p 00010000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f6d90a96000-7f6d90a97000 r--p 0000f000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f6d90a97000-7f6d90a98000 rw-p 00010000 fe:06 400922                     /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
      7f6d90a98000-7f6d90a9f000 r-xp 00000000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f6d90a9f000-7f6d90c9f000 ---p 00007000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f6d90c9f000-7f6d90ca0000 r--p 00007000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f6d90ca0000-7f6d90ca1000 rw-p 00008000 fe:06 402403                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
      7f6d90ca1000-7f6d90cc1000 r-xp 00000000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f6d90cc1000-7f6d90ec0000 ---p 00020000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f6d90ec0000-7f6d90ec1000 r--p 0001f000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f6d90ec1000-7f6d90ec5000 rw-p 00020000 fe:06 400915                     /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
      7f6d90ec5000-7f6d90ed9000 r-xp 00000000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f6d90ed9000-7f6d910d8000 ---p 00014000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f6d910d8000-7f6d910d9000 r--p 00013000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f6d910d9000-7f6d910da000 rw-p 00014000 fe:06 394205                     /usr/lib/x86_64-linux-gnu/libresolv-2.24.so
      7f6d910da000-7f6d910dc000 rw-p 00000000 00:00 0 
      7f6d910dc000-7f6d911df000 r-xp 00000000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f6d911df000-7f6d913de000 ---p 00103000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f6d913de000-7f6d913df000 r--p 00102000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f6d913df000-7f6d913e0000 rw-p 00103000 fe:06 394157                     /usr/lib/x86_64-linux-gnu/libm-2.24.so
      7f6d913e0000-7f6d913e2000 r-xp 00000000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f6d913e2000-7f6d915e1000 ---p 00002000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f6d915e1000-7f6d915e2000 r--p 00001000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f6d915e2000-7f6d915e3000 rw-p 00002000 fe:06 394230                     /usr/lib/x86_64-linux-gnu/libutil-2.24.so
      7f6d915e3000-7f6d915fc000 r-xp 00000000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f6d915fc000-7f6d917fb000 ---p 00019000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f6d917fb000-7f6d917fc000 r--p 00018000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f6d917fc000-7f6d917fd000 rw-p 00019000 fe:06 394235                     /usr/lib/x86_64-linux-gnu/libz.so.1.2.8
      7f6d917fd000-7f6d91817000 r-xp 015: test_drain_support_1_all_messages (system_tests_drain.DrainSupportTest) ... ok
      0000000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f6d91817000-7f6d91a16000 ---p 0001a000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f6d91a16000-7f6d91a17000 r--p 00019000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f6d91a17000-7f6d91a18000 rw-p 0001a000 fe:06 402611                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
      7f6d91a18000-7f6d91c4d000 r-xp 00000000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f6d91c4d000-7f6d91e4d000 ---p 00235000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f6d91e4d000-7f6d91e69000 r--p 00235000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f6d91e69000-7f6d91e78000 rw-p 00251000 fe:06 402359                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
      7f6d91e78000-7f6d91e7c000 rw-p 00000000 00:00 0 
      7f6d91e7c000-7f6d91edb000 r-xp 00000000 fe:06 402627                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
      7f6d91edb000-7f6d920db000 ---p 0005f000 fe:06 402627                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
      7f6d920db000-7f6d920df000 r--p 0005f000 fe:06 402627                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
      7f6d920df000-7f6d920e5000 rw-p 00063000 fe:06 402627                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2
      7f6d920e5000-7f6d9227a000 r-xp 00000000 fe:06 394121                     /usr/lib/x86_64-linux-gnu/libc-2.24.so
      7f6d9227a000-7f6d92479000 ---p 00195000 fe:06 394121                     /usr/lib/x86_64-linux-gnu/libc-2.24.so
      7f6d92479000-7f6d9247d000 r--p 00194000 fe:06 394121                     /usr/lib/x86_64-linux-gnu/libc-2.24.so
      7f6d9247d000-7f6d9247f000 rw-p 00198000 fe:06 394121                     /usr/lib/x86_64-linux-gnu/libc-2.24.so
      7f6d9247f000-7f6d92483000 rw-p 00000000 00:00 0 
      7f6d92483000-7f6d92778000 r-xp 00000000 fe:06 402591                     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
      7f6d92778000-7f6d92978000 ---p 002f5000 fe:06 402591                     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
      7f6d92978000-7f6d9297a000 r--p 002f5000 fe:06 402591                     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
      7f6d9297a000-7f6d929f1000 rw-p 002f7000 fe:06 402591                     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
      7f6d929f1000-7f6d92a15000 rw-p 00000000 00:00 0 
      7f6d92a15000-7f6d92a18000 r-xp 00000000 fe:06 394134                     /usr/lib/x86_64-linux-gnu/libdl-2.24.so
      7f6d92a18000-7f6d92c17000 ---p 00003000 fe:06 394134                     /usr/lib/x86_64-linux-gnu/libdl-2.24.so
      7f6d92c17000-7f6d92c18000 r--p 00002000 fe:06 394134                     /usr/lib/x86_64-linux-gnu/libdl-2.24.so
      7f6d92c18000-7f6d92c19000 rw-p 00003000 fe:06 394134                     /usr/lib/x86_64-linux-gnu/libdl-2.24.so
      7f6d92c19000-7f6d92c20000 r-xp 00000000 fe:06 394207                     /usr/lib/x86_64-linux-gnu/librt-2.24.so
      7f6d92c20000-7f6d92e1f000 ---p 00007000 fe:06 394207                     /usr/lib/x86_64-linux-gnu/librt-2.24.so
      7f6d92e1f000-7f6d92e20000 r--p 00006000 fe:06 394207                     /usr/lib/x86_64-linux-gnu/librt-2.24.so
      7f6d92e20000-7f6d92e21000 rw-p 00007000 fe:06 394207                     /usr/lib/x86_64-linux-gnu/librt-2.24.so
      7f6d92e21000-7f6d92e39000 r-xp 00000000 fe:06 394203                     /usr/lib/x86_64-linux-gnu/libpthread-2.24.so
      7f6d92e39000-7f6d93038000 ---p 00018000 fe:06 394203                     /usr/lib/x86_64-linux-gnu/libpthread-2.24.so
      7f6d93038000-7f6d93039000 r--p 00017000 fe:06 394203                     /usr/lib/x86_64-linux-gnu/libpthread-2.24.so
      7f6d93039000-7f6d9303a000 rw-p 00018000 fe:06 394203                     /usr/lib/x86_64-linux-gnu/libpthread-2.24.so
      7f6d9303a000-7f6d9303e000 rw-p 00000000 00:00 0 
      7f6d9303e000-7f6d93048000 r-xp 00000000 fe:06 412321                     /usr/lib/libqpid-proton-proactor.so.11.0.0
      7f6d93048000-7f6d93247000 ---p 0000a000 fe:06 412321                     /usr/lib/libqpid-proton-proactor.so.11.0.0
      7f6d93247000-7f6d93248000 r--p 00009000 fe:06 412321                     /usr/lib/libqpid-proton-proactor.so.11.0.0
      7f6d93248000-7f6d93249000 rw-p 0000a000 fe:06 412321                     /usr/lib/libqpid-proton-proactor.so.11.0.0
      7f6d93249000-7f6d9328f000 r-xp 00000000 fe:06 412318                     /usr/lib/libqpid-proton-core.so.11.0.0
      7f6d9328f000-7f6d9348e000 ---p 00046000 fe:06 412318                     /usr/lib/libqpid-proton-core.so.11.0.0
      7f6d9348e000-7f6d93490000 r--p 00045000 fe:06 412318                     /usr/lib/libqpid-proton-core.so.11.0.0
      7f6d93490000-7f6d93491000 rw-p 00047000 fe:06 412318                     /usr/lib/libqpid-proton-core.so.11.0.0
      7f6d93491000-7f6d93500000 r-xp 00000000 fe:06 537648                     /main/qpid-dispatch/build/src/libqpid-dispatch.so
      7f6d93500000-7f6d93700000 ---p 0006f000 fe:06 537648                     /main/qpid-dispatch/build/src/libqpid-dispatch.so
      7f6d93700000-7f6d93701000 r--p 0006f000 fe:06 537648                     /main/qpid-dispatch/build/src/libqpid-dispatch.so
      7f6d93701000-7f6d93706000 rw-p 00070000 fe:06 537648                     /main/qpid-dispatch/build/src/libqpid-dispatch.so
      7f6d93706000-7f6d93707000 rw-p 00000000 00:00 0 
      7f6d93707000-7f6d9372a000 r-xp 00000000 fe:06 394099                     /usr/lib/x86_64-linux-gnu/ld-2.24.so
      7f6d93766000-7f6d93826000 rw-p 00000000 00:00 0 
      7f6d93857000-7f6d93920000 rw-p 00000000 00:00 0 
      7f6d93925000-7f6d93926000 rw-p 00000000 00:00 0 
      7f6d93926000-7f6d93927000 rwxp 00000000 00:00 0 
      7f6d93927000-7f6d9392a000 rw-p 00000000 00:00 0 
      7f6d9392a000-7f6d9392b000 r--p 00023000 fe:06 394099                     /usr/lib/x86_64-linux-gnu/ld-2.24.so
      7f6d9392b000-7f6d9392c000 rw-p 00024000 fe:06 394099                     /usr/lib/x86_64-linux-gnu/ld-2.24.so
      7f6d9392c000-7f6d9392d000 rw-p 00000000 00:00 0 
      7ffcf3fd9000-7ffcf3ffa000 rw-p 00000000 00:00 0                          [stack]
      7ffcf3ffa000-7ffcf3ffc000 r--p 00000000 00:00 0                          [vvar]
      7ffcf3ffc000-7ffcf3ffe000 r-xp 00000000 00:00 0                          [vdso]
      ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
      15: test_drain_support_2_one_message (system_tests_drain.DrainSupportTest) ... ERROR:root:proton:io: recv: Connection reset by peer
      15: ERROR:root:proton:io: recv: Connection reset by peer
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: FAIL
      15: test_drain_support_3_no_messages (system_tests_drain.DrainSupportTest) ... ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: FAIL
      15: test_drain_support_4_no_more_messages (system_tests_drain.DrainSupportTest) ... ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: ERROR:root:proton:io: recv: Connection refused
      15: FAIL
      15: ERROR
      15: 
      15: ======================================================================
      15: ERROR: tearDownClass (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_test.py", line 605, in tearDownClass
      15:     cls.tester.teardown()
      15:   File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
      15:     raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
      15: RuntimeError: Errors during teardown: 
      15: 
      15: Process 2032 error: exit code -6, expected -1
      15: qdrouterd -c test-router.conf -I /main/qpid-dispatch/python
      15: /main/qpid-dispatch/build/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/test-router-1.cmd
      15: >>>>
      15: <<<<
      15: 
      15: ======================================================================
      15: FAIL: test_drain_support_2_one_message (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_tests_drain.py", line 53, in test_drain_support_2_one_message
      15:     self.assertEqual(drain_support.error, None)
      15: AssertionError: 'Timeout Expired: sent: 0 rcvd: 0' != None
      15: 
      15: ======================================================================
      15: FAIL: test_drain_support_3_no_messages (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_tests_drain.py", line 58, in test_drain_support_3_no_messages
      15:     self.assertEqual(drain_support.error, None)
      15: AssertionError: 'Timeout Expired' != None
      15: 
      15: ======================================================================
      15: FAIL: test_drain_support_4_no_more_messages (system_tests_drain.DrainSupportTest)
      15: ----------------------------------------------------------------------
      15: Traceback (most recent call last):
      15:   File "/main/qpid-dispatch/tests/system_tests_drain.py", line 63, in test_drain_support_4_no_more_messages
      15:     self.assertEqual(drain_support.error, None)
      15: AssertionError: 'Timeout Expired: sent=0 rcvd=0' != None
      15: 
      15: ----------------------------------------------------------------------
      15: Ran 4 tests in 188.308s
      15: 
      15: FAILED (failures=3, errors=1)
          Test #15: system_tests_drain ...............***Failed  188.35 sec
      
      0% tests passed, 1 tests failed out of 1
      
      Total Test time (real) = 190.91 sec
      
      The following tests FAILED:
               15 - system_tests_drain (Failed)
      Errors while running CTest
      root@519997eb8015:/main/qpid-dispatch/build# find / -name core
      /main/qpid-dispatch/build/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/core
      /main/qpid-proton/build/proton-c/CMakeFiles/qpid-proton-core.dir/src/core
      /main/qpid-proton/build/proton-c/CMakeFiles/qpid-proton.dir/src/core
      /main/qpid-proton/build/proton-c/bindings/python/dist/proton-c/src/core
      /main/qpid-proton/proton-c/src/core
      /main/qpid-proton/proton-c/bindings/ruby/lib/core
      /proc/sys/net/core
      /dev/core
      
      root@519997eb8015:/main/qpid-dispatch/build# gdb -c /main/qpid-dispatch/build/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/core `which qdrouterd`
      GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
      Copyright (C) 2016 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      Type "show configuration" for configuration details.
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>.
      Find the GDB manual and other documentation resources online at:
      <http://www.gnu.org/software/gdb/documentation/>.
      For help, type "help".
      Type "apropos word" to search for commands related to "word"...
      Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols found)...done.
      [New LWP 2034]
      [New LWP 2033]
      [New LWP 2036]
      [New LWP 2032]
      [New LWP 2035]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `qdrouterd -c test-router.conf -I /main/qpid-dispatch/python'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
      [Current thread is 1 (Thread 0x7f6d8f167700 (LWP 2034))]
      (gdb) thread apply all bt
      
      Thread 5 (Thread 0x7f6d8e966700 (LWP 2035)):
      #0  0x00007f6d921cdf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f6d93044fcd in proactor_do_epoll (p=0x557e3498dc20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f6d93045185 in pn_proactor_wait (p=0x557e3498dc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f6d934e6e7a in thread_run (arg=0x557e3475e370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f6d92e28494 in start_thread (arg=0x7f6d8e966700) at pthread_create.c:333
      #5  0x00007f6d921cd93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 4 (Thread 0x7f6d93918080 (LWP 2032)):
      #0  0x00007f6d921cdf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f6d93044fcd in proactor_do_epoll (p=0x557e3498dc20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f6d93045185 in pn_proactor_wait (p=0x557e3498dc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f6d934e6e7a in thread_run (arg=0x557e3475e370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f6d934e7987 in qd_server_run (qd=0x557e34702010) at /main/qpid-dispatch/src/server.c:1031
      #5  0x0000557e3390b8d0 in ?? ()
      #6  0x0000557e3390c25b in main ()
      
      Thread 3 (Thread 0x7f6d8e165700 (LWP 2036)):
      #0  0x00007f6d921cdf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
      #1  0x00007f6d93044fcd in proactor_do_epoll (p=0x557e3498dc20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1636
      #2  0x00007f6d93045185 in pn_proactor_wait (p=0x557e3498dc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #3  0x00007f6d934e6e7a in thread_run (arg=0x557e3475e370) at /main/qpid-dispatch/src/server.c:817
      #4  0x00007f6d92e28494 in start_thread (arg=0x7f6d8e165700) at pthread_create.c:333
      #5  0x00007f6d921cd93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 2 (Thread 0x7f6d8f968700 (LWP 2033)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      #1  0x00007f6d934c49c0 in sys_cond_wait (cond=0x557e34a045c0, held_mutex=0x557e34a04680)
          at /main/qpid-dispatch/src/posix/threading.c:91
      #2  0x00007f6d934da0fa in router_core_thread (arg=0x557e34a042b0) at /main/qpid-dispatch/src/router_core/router_core_thread.c:53
      #3  0x00007f6d92e28494 in start_thread (arg=0x7f6d8f968700) at pthread_create.c:333
      #4  0x00007f6d921cd93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      
      Thread 1 (Thread 0x7f6d8f167700 (LWP 2034)):
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
      #1  0x00007f6d921193fa in __GI_abort () at abort.c:89
      #2  0x00007f6d92155bd0 in __libc_message (do_abort=do_abort@entry=2, 
          fmt=fmt@entry=0x7f6d9224abd0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
      #3  0x00007f6d9215bf96 in malloc_printerr (action=3, str=0x7f6d9224ac48 "double free or corruption (!prev)", ptr=<optimized out>, 
          ar_ptr=<optimized out>) at malloc.c:5046
      #4  0x00007f6d9215c78e in _int_free (av=0x7f6d80000020, p=0x7f6d80013350, have_lock=0) at malloc.c:3902
      #5  0x00007f6d93258289 in pn_object_free (object=0x7f6d80013370) at /main/qpid-proton/proton-c/src/core/object/object.c:243
      #6  0x00007f6d93257d8a in pn_class_decref (clazz=0x7f6d93247c20 <pconnection_class>, object=0x7f6d80013370)
          at /main/qpid-proton/proton-c/src/core/object/object.c:101
      #7  0x00007f6d932582d2 in pn_decref (object=0x7f6d80013370) at /main/qpid-proton/proton-c/src/core/object/object.c:253
      #8  0x00007f6d930422f0 in pconnection_final_free (pc=0x7f6d80013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:639
      #9  0x00007f6d93042395 in pconnection_cleanup (pc=0x7f6d80013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:653
      #10 0x00007f6d93042ac5 in pconnection_process (pc=0x7f6d80013370, events=0, timeout=false, topup=false)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:845
      #11 0x00007f6d930450e5 in proactor_do_epoll (p=0x557e3498dc20, can_block=true)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1664
      #12 0x00007f6d93045185 in pn_proactor_wait (p=0x557e3498dc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      #13 0x00007f6d934e6e7a in thread_run (arg=0x557e3475e370) at /main/qpid-dispatch/src/server.c:817
      #14 0x00007f6d92e28494 in start_thread (arg=0x7f6d8f167700) at pthread_create.c:333
      #15 0x00007f6d921cd93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      (gdb) bt full
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
              set = {__val = {0, 4135490170470932528, 3256155514231470387, 3618980097156343351, 3276212736508309601, 3472328296227676272, 
                  3472339291342909488, 7221071439114285088, 3472328506715222841, 4121701186546317101, 3274715270390768178, 
                  3689064028293853229, 3475201551770857520, 4123379934690484278, 2314885530818453536, 2314885530818453536}}
              pid = <optimized out>
              tid = <optimized out>
      #1  0x00007f6d921193fa in __GI_abort () at abort.c:89
              save_stage = 2
              act = {__sigaction_handler = {sa_handler = 0x6b636174735b2020, sa_sigaction = 0x6b636174735b2020}, sa_mask = {__val = {
                    3703757035011508829, 3975887029566269030, 7162524628326508134, 8083166874389458992, 3472328296227680288, 
                    2319406834570502192, 2314885530818453552, 2314885530818453536, 2314885530818453536, 8241999043771768864, 
                    3703757035011508829, 3975887029566400102, 7306639816402364006, 8104277497642758192, 3472328296227680288, 
                    140108528771856}}, sa_flags = 84, sa_restorer = 0x7f6d8f166310}
              sigs = {__val = {32, 0 <repeats 15 times>}}
      #2  0x00007f6d92155bd0 in __libc_message (do_abort=do_abort@entry=2, 
          fmt=fmt@entry=0x7f6d9224abd0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
              ap = {{gp_offset = 40, fp_offset = 1, overflow_arg_area = 0x7f6d8f166320, reg_save_area = 0x7f6d8f1662b0}}
              fd = 9
              on_2 = <optimized out>
              list = <optimized out>
              nlist = <optimized out>
              cp = <optimized out>
              written = <optimized out>
      #3  0x00007f6d9215bf96 in malloc_printerr (action=3, str=0x7f6d9224ac48 "double free or corruption (!prev)", ptr=<optimized out>, 
          ar_ptr=<optimized out>) at malloc.c:5046
              buf = "00007f6d80013360"
              cp = <optimized out>
              ar_ptr = <optimized out>
              ptr = <optimized out>
              str = 0x7f6d9224ac48 "double free or corruption (!prev)"
              action = 3
      #4  0x00007f6d9215c78e in _int_free (av=0x7f6d80000020, p=0x7f6d80013350, have_lock=0) at malloc.c:3902
              size = <optimized out>
              fb = <optimized out>
              nextchunk = <optimized out>
              nextsize = <optimized out>
              nextinuse = <optimized out>
              prevsize = <optimized out>
              bck = <optimized out>
              fwd = <optimized out>
              errstr = <optimized out>
              locked = <optimized out>
              __func__ = "_int_free"
      #5  0x00007f6d93258289 in pn_object_free (object=0x7f6d80013370) at /main/qpid-proton/proton-c/src/core/object/object.c:243
              head = 0x7f6d80013360
      #6  0x00007f6d93257d8a in pn_class_decref (clazz=0x7f6d93247c20 <pconnection_class>, object=0x7f6d80013370)
          at /main/qpid-proton/proton-c/src/core/object/object.c:101
              rc = 0
              __PRETTY_FUNCTION__ = "pn_class_decref"
      #7  0x00007f6d932582d2 in pn_decref (object=0x7f6d80013370) at /main/qpid-proton/proton-c/src/core/object/object.c:253
      No locals.
      #8  0x00007f6d930422f0 in pconnection_final_free (pc=0x7f6d80013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:639
      No locals.
      #9  0x00007f6d93042395 in pconnection_cleanup (pc=0x7f6d80013370) at /main/qpid-proton/proton-c/src/proactor/epoll.c:653
              can_free = true
      #10 0x00007f6d93042ac5 in pconnection_process (pc=0x7f6d80013370, events=0, timeout=false, topup=false)
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:845
              inbound_wake = false
              timer_unarmed = false
              timer_fired = false
              waking = false
              tick_required = false
              __PRETTY_FUNCTION__ = "pconnection_process"
              closed = false
              unarmed = false
              rearm = false
      #11 0x00007f6d930450e5 in proactor_do_epoll (p=0x557e3498dc20, can_block=true)
      ---Type <return> to continue, or q <return> to quit---
          at /main/qpid-proton/proton-c/src/proactor/epoll.c:1664
              pc = 0x7f6d80013370
              batch = 0x0
              ev = {events = 4, data = {ptr = 0x7f6d80013380, fd = -2147404928, u32 = 2147562368, u64 = 140108275725184}}
              n = 1
              ee = 0x7f6d80013380
              timeout = -1
              __PRETTY_FUNCTION__ = "proactor_do_epoll"
      #12 0x00007f6d93045185 in pn_proactor_wait (p=0x557e3498dc20) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1682
      No locals.
      #13 0x00007f6d934e6e7a in thread_run (arg=0x557e3475e370) at /main/qpid-dispatch/src/server.c:817
              events = 0x7f6d800138c0
              e = 0x0
              qd_server = 0x557e3475e370
              running = true
      #14 0x00007f6d92e28494 in start_thread (arg=0x7f6d8f167700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7f6d8f167700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140108528776960, -754696317570653812, 0, 140724402099327, 0, 140108604026944, 
                      819049981324812684, 819059710486241676}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {
                    prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #15 0x00007f6d921cd93f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
      No locals.
      (gdb) 
      

      Attachments

        1. DISPATCH-777_3.core.zip
          1.89 MB
          Jiri Daněk
        2. DISPATCH-777_2.core.zip
          1.90 MB
          Jiri Daněk
        3. DISPATCH-777_1.core.zip
          1.89 MB
          Jiri Daněk

        Issue Links

          Activity

            People

              aconway Alan Conway
              jdanek Jiri Daněk
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: