[DISPATCH-2127] message content races between readers and writers

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.16.0
• Fix Version/s: 1.20.0
• Component/s: Router Node
• Labels: race-condition tsan
• Assignee: Ken Giusti
• Reporter: Ken Giusti
• Created: 17/May/21 21:32
• Updated: 15/Mar/22 21:54

Description

TSAN complains about races between the sending and receiving threads accessing the message's context fields:

 

content->aborted flag

content->qd_input_holdoff

 

65: WARNING: ThreadSanitizer: data race (pid=569024)
65: Write of size 1 at 0x7b64000bdab4 by thread T3 (mutexes: write M381533902092816192):
65: #0 qd_message_receive /home/kgiusti/work/dispatch/qpid-dispatch/src/message.c:1533 (libqpid-dispatch.so+0x93b63)
65: #1 AMQP_rx_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:466 (libqpid-dispatch.so+0x1233ed)
65: #2 do_receive /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:227 (libqpid-dispatch.so+0x7a99b)
65: #3 qd_container_handle_event /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:737 (libqpid-dispatch.so+0x7cf4f)
65: #4 handle /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1096 (libqpid-dispatch.so+0x130537)
65: #5 thread_run /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1121 (libqpid-dispatch.so+0x13063a)
65: #6 _thread_init /home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c:172 (libqpid-dispatch.so+0xad37a)
65: #7 <null> <null> (libtsan.so.0+0x2d33f)
65:
65: Previous read of size 1 at 0x7b64000bdab4 by thread T2:
65: #0 qd_message_send /home/kgiusti/work/dispatch/qpid-dispatch/src/message.c:1885 (libqpid-dispatch.so+0x95b5d)
65: #1 CORE_link_deliver /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:1985 (libqpid-dispatch.so+0x127f9f)
65: #2 qdr_link_process_deliveries /home/kgiusti/work/dispatch/qpid-dispatch/src/router_core/transfer.c:178 (libqpid-dispatch.so+0x1045c6)
65: #3 CORE_link_push /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:1920 (libqpid-dispatch.so+0x127d00)
65: #4 qdr_connection_process /home/kgiusti/work/dispatch/qpid-dispatch/src/router_core/connections.c:414 (libqpid-dispatch.so+0xc4bec)
65: #5 AMQP_writable_conn_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:299 (libqpid-dispatch.so+0x122d42)
65: #6 writable_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:395 (libqpid-dispatch.so+0x7b2e2)
65: #7 qd_container_handle_event /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:747 (libqpid-dispatch.so+0x7cfd5)
65: #8 handle /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1096 (libqpid-dispatch.so+0x130537)
65: #9 thread_run /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1121 (libqpid-dispatch.so+0x13063a)
65: #10 _thread_init /home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c:172 (libqpid-dispatch.so+0xad37a)
65: #11 <null> <null> (libtsan.so.0+0x2d33f)

 

65: ==================
65: WARNING: ThreadSanitizer: data race (pid=569024)
65: Write of size 1 at 0x7b6400057cb4 by thread T3 (mutexes: write M49320):
65: #0 qd_message_receive /home/kgiusti/work/dispatch/qpid-dispatch/src/message.c:1533 (libqpid-dispatch.so+0x93b63)
65: #1 AMQP_rx_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:466 (libqpid-dispatch.so+0x1233ed)
65: #2 deferred_AMQP_rx_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:913 (libqpid-dispatch.so+0x1245cb)
65: #3 invoke_deferred_calls /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:789 (libqpid-dispatch.so+0x12eba3)
65: #4 handle /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1046 (libqpid-dispatch.so+0x12ffa5)
65: #5 thread_run /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1121 (libqpid-dispatch.so+0x13063a)
65: #6 _thread_init /home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c:172 (libqpid-dispatch.so+0xad37a)
65: #7 <null> <null> (libtsan.so.0+0x2d33f)
65:
65: Previous read of size 1 at 0x7b6400057cb4 by thread T2:
65: #0 qd_message_send /home/kgiusti/work/dispatch/qpid-dispatch/src/message.c:1984 (libqpid-dispatch.so+0x95bdb)
65: #1 CORE_link_deliver /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:1985 (libqpid-dispatch.so+0x127f9f)
65: #2 qdr_link_process_deliveries /home/kgiusti/work/dispatch/qpid-dispatch/src/router_core/transfer.c:178 (libqpid-dispatch.so+0x1045c6)
65: #3 CORE_link_push /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:1920 (libqpid-dispatch.so+0x127d00)
65: #4 qdr_connection_process /home/kgiusti/work/dispatch/qpid-dispatch/src/router_core/connections.c:414 (libqpid-dispatch.so+0xc4bec)
65: #5 AMQP_writable_conn_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:299 (libqpid-dispatch.so+0x122d42)
65: #6 writable_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:395 (libqpid-dispatch.so+0x7b2e2)
65: #7 qd_container_handle_event /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:747 (libqpid-dispatch.so+0x7cfd5)
65: #8 handle /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1096 (libqpid-dispatch.so+0x130537)
65: #9 thread_run /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1121 (libqpid-dispatch.so+0x13063a)
65: #10 _thread_init /home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c:172 (libqpid-dispatch.so+0xad37a)
65: #11 <null> <null> (libtsan.so.0+0x2d33f)

 

5: WARNING: ThreadSanitizer: data race (pid=569024)
65: Read of size 1 at 0x7b640005ccb3 by main thread:
65: #0 qd_message_receive /home/kgiusti/work/dispatch/qpid-dispatch/src/message.c:1493 (libqpid-dispatch.so+0x93714)
65: #1 AMQP_rx_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:466 (libqpid-dispatch.so+0x1233ed)
65: #2 do_receive /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:227 (libqpid-dispatch.so+0x7a99b)
65: #3 qd_container_handle_event /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:737 (libqpid-dispatch.so+0x7cf4f)
65: #4 handle /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1096 (libqpid-dispatch.so+0x130537)
65: #5 thread_run /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1121 (libqpid-dispatch.so+0x13063a)
65: #6 qd_server_run /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1501 (libqpid-dispatch.so+0x1328de)
65: #7 main_process /home/kgiusti/work/dispatch/qpid-dispatch/router/src/main.c:115 (qdrouterd+0x40271a)
65: #8 main /home/kgiusti/work/dispatch/qpid-dispatch/router/src/main.c:369 (qdrouterd+0x403557)
65:
65: Previous write of size 1 at 0x7b640005ccb3 by thread T2 (mutexes: write M827390265202469824):
65: #0 qd_message_send /home/kgiusti/work/dispatch/qpid-dispatch/src/message.c:1952 (libqpid-dispatch.so+0x959a6)
65: #1 CORE_link_deliver /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:1985 (libqpid-dispatch.so+0x127f9f)
65: #2 qdr_link_process_deliveries /home/kgiusti/work/dispatch/qpid-dispatch/src/router_core/transfer.c:178 (libqpid-dispatch.so+0x1045c6)
65: #3 CORE_link_push /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:1920 (libqpid-dispatch.so+0x127d00)
65: #4 qdr_connection_process /home/kgiusti/work/dispatch/qpid-dispatch/src/router_core/connections.c:414 (libqpid-dispatch.so+0xc4bec)
65: #5 AMQP_writable_conn_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/router_node.c:299 (libqpid-dispatch.so+0x122d42)
65: #6 writable_handler /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:395 (libqpid-dispatch.so+0x7b2e2)
65: #7 qd_container_handle_event /home/kgiusti/work/dispatch/qpid-dispatch/src/container.c:747 (libqpid-dispatch.so+0x7cfd5)
65: #8 handle /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1096 (libqpid-dispatch.so+0x130537)
65: #9 thread_run /home/kgiusti/work/dispatch/qpid-dispatch/src/server.c:1121 (libqpid-dispatch.so+0x13063a)
65: #10 _thread_init /home/kgiusti/work/dispatch/qpid-dispatch/src/posix/threading.c:172 (libqpid-dispatch.so+0xad37a)
65: #11 <null> <null> (libtsan.so.0+0x2d33f)