[DISPATCH-2048] system_tests_router_mesh failure due to leak of qd_deferred_call_t

• Type: Test
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.16.0
• Component/s: Tests
• Assignee: Ganesh Murthy
• Reporter: Ganesh Murthy
• Created: 15/Apr/21 16:18
• Updated: 14/May/21 14:09
• Resolved: 19/Apr/21 14:17

Description

 63: Test command: /usr/bin/python "/home/travis/build/apache/qpid-dispatch/build/tests/run.py" "-m" "unittest" "-v" "system_tests_router_mesh"
63: Test timeout computed to be: 600
63: test_01_rx_failover_clean (system_tests_router_mesh.ThreeRouterTest) ... ok
63: test_02_rx_failover_dirty (system_tests_router_mesh.ThreeRouterTest) ... ok
63: test_03_unavailable_link_attach (system_tests_router_mesh.ThreeRouterTest) ... ok
63: test_04_unavailable_anonymous_link_attach (system_tests_router_mesh.ThreeRouterTest) ... ok
63: test_05_unavailable_anonymous_link_send (system_tests_router_mesh.ThreeRouterTest) ... ok
63: test_06_parallel_priority (system_tests_router_mesh.ThreeRouterTest) ... ERROR
63: ERROR
63: 
63: Router RouterC output file:
63: ERROR: Aborted due to unexpected alloc pool leak of type 'qd_deferred_call_t'
63: 
63: <<<<
63: 
63: Router RouterC debug dump file:
63: >>>>
63: alloc.c: Items of type 'qd_iterator_t' remain allocated at shutdown: 80 (SUPPRESSED)
63: alloc.c: Items of type 'qd_timer_t' remain allocated at shutdown: 1 (SUPPRESSED)
63: alloc.c: Items of type 'qd_bitmask_t' remain allocated at shutdown: 8 (SUPPRESSED)
63: alloc.c: Items of type 'qd_buffer_t' remain allocated at shutdown: 287 (SUPPRESSED)
63: alloc.c: Items of type 'qd_parsed_field_t' remain allocated at shutdown: 32 (SUPPRESSED)
63: alloc.c: Items of type 'qd_connector_t' remain allocated at shutdown: 1 (SUPPRESSED)
63: alloc.c: Items of type 'qd_message_t' remain allocated at shutdown: 8 (SUPPRESSED)
63: alloc.c: Items of type 'qd_message_content_t' remain allocated at shutdown: 8 (SUPPRESSED)
63: alloc.c: Items of type 'qdr_delivery_t' remain allocated at shutdown: 8 (SUPPRESSED)
63: alloc.c: Items of type 'qd_link_ref_t' remain allocated at shutdown: 8 (SUPPRESSED)
63: alloc.c: Items of type 'qd_deferred_call_t' remain allocated at shutdown: 15
63: Leak: 2021-04-15 03:43:23.586342 +0000 type: qd_deferred_call_t address: 0x6110001b7d10
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x53431e) [0x7f3be6a9231e]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qdr_connection_process+0x2720) [0x7f3be69b31d4]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x52a9c7) [0x7f3be6a889c7]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x3b631c) [0x7f3be691431c]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_container_handle_event+0x223b) [0x7f3be6917d6a]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x54621b) [0x7f3be6aa421b]
63: 
63: Leak: 2021-04-15 03:43:23.586520 +0000 type: qd_deferred_call_t address: 0x61100014b9d0
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x53431e) [0x7f3be6a9231e]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qdr_connection_process+0x2720) [0x7f3be69b31d4]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x52a9c7) [0x7f3be6a889c7]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x3b631c) [0x7f3be691431c]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_container_handle_event+0x223b) [0x7f3be6917d6a]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x54621b) [0x7f3be6aa421b]
63: 
63: Leak: 2021-04-15 03:43:23.586694 +0000 type: qd_deferred_call_t address: 0x6110001ad950
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x53431e) [0x7f3be6a9231e]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qdr_connection_process+0x2720) [0x7f3be69b31d4]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x52a9c7) [0x7f3be6a889c7]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x3b631c) [0x7f3be691431c]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_container_handle_event+0x223b) [0x7f3be6917d6a]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x54621b) [0x7f3be6aa421b]
63: 
63: Leak: 2021-04-15 03:43:23.586872 +0000 type: qd_deferred_call_t address: 0x6110001a6650
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x53431e) [0x7f3be6a9231e]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qdr_connection_process+0x2720) [0x7f3be69b31d4]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x52a9c7) [0x7f3be6a889c7]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x3b631c) [0x7f3be691431c]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_container_handle_event+0x223b) [0x7f3be6917d6a]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x54621b) [0x7f3be6aa421b]
63: 
63: Leak: 2021-04-15 03:43:23.587044 +0000 type: qd_deferred_call_t address: 0x6110001b7950
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x53431e) [0x7f3be6a9231e]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qdr_connection_process+0x2720) [0x7f3be69b31d4]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x52a9c7) [0x7f3be6a889c7]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x3b631c) [0x7f3be691431c]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_container_handle_event+0x223b) [0x7f3be6917d6a]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x54621b) [0x7f3be6aa421b]
63: 
63: Leak: 2021-04-15 03:43:23.587219 +0000 type: qd_deferred_call_t address: 0x6110001ad810
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x53431e) [0x7f3be6a9231e]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qdr_connection_process+0x2720) [0x7f3be69b31d4]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x52a9c7) [0x7f3be6a889c7]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x3b631c) [0x7f3be691431c]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_container_handle_event+0x223b) [0x7f3be6917d6a]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x54621b) [0x7f3be6aa421b]
63: 
63: Leak: 2021-04-15 03:43:23.587391 +0000 type: qd_deferred_call_t address: 0x6110001ad6d0
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x53431e) [0x7f3be6a9231e]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qdr_connection_process+0x2720) [0x7f3be69b31d4]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x52a9c7) [0x7f3be6a889c7]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x3b631c) [0x7f3be691431c]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_container_handle_event+0x223b) [0x7f3be6917d6a]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x54621b) [0x7f3be6aa421b]
63: 
63: Leak: 2021-04-15 03:43:23.587568 +0000 type: qd_deferred_call_t address: 0x61100014ad50
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x53431e) [0x7f3be6a9231e]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qdr_connection_process+0x2720) [0x7f3be69b31d4]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x52a9c7) [0x7f3be6a889c7]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x3b631c) [0x7f3be691431c]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_container_handle_event+0x223b) [0x7f3be6917d6a]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x54621b) [0x7f3be6aa421b]
63: 
63: Leak: 2021-04-15 03:43:23.602923 +0000 type: qd_deferred_call_t address: 0x61100014a350
63: /usr/lib/x86_64-linux-gnu/libasan.so.2(+0x4a077) [0x7f3be716c077]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_alloc+0xc70) [0x7f3be68ee7a0]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(new_qd_deferred_call_t+0x26) [0x7f3be6a97777]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_connection_invoke_deferred+0x25) [0x7f3be6aaa479]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_link_q2_restart_receive+0x102) [0x7f3be6a92d5f]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_message_free+0xdaf) [0x7f3be6945621]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x4d0b1b) [0x7f3be6a2eb1b]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x4cc180) [0x7f3be6a2a180]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(qd_timer_visit+0xf6) [0x7f3be6ab020d]
63: /home/travis/build/apache/qpid-dispatch/build/src/libqpid-dispatch.so(+0x544eef) [0x7f3be6aa2eef]

Issue Links

• is duplicated by: DISPATCH-1858 HTTP1: occasional failure with leak of qd_deferred_call_t item