Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
1.13.0, 1.14.0
-
None
Description
While we dig into the issue below I thought it worth reaching out to see if this is a familiar issue or in code that has changed much since 1.13 so may have likely already been fixed.
We have a freebsd product that is using qdrouterd. We have had some crashes during closing of connections to a broker so built a qdrouterd with asan on. This is hitting the use-after-free issue in uv__run_closing_handles below fairly reliably.
It appears from first inspection that uv__finish_close is trying to remove a handle from the handle_queue at line 300 but the handle_queue has reference into a pconnection_t that has already been freed.
Is this an area of code that has had issues in the past? Is this likely to have been fixed in the latest release? Any suggestions on where to add printfs etc to identify the culprit?
243static void uv__finish_close(uv_handle_t* handle) {
244 uv_signal_t* sh;
245
246 /* Note: while the handle is in the UV_HANDLE_CLOSING state now, it's still
247 * possible for it to be active in the sense that uv__is_active() returns
248 * true.
249 *
250 * A good example is when the user calls uv_shutdown(), immediately followed
251 * by uv_close(). The handle is considered active at this point because the
252 * completion of the shutdown req is still pending.
253 */
254 assert(handle->flags & UV_HANDLE_CLOSING);
255 assert(!(handle->flags & UV_HANDLE_CLOSED));
256 handle->flags |= UV_HANDLE_CLOSED;
257
258 switch (handle->type) {
259 case UV_PREPARE:
260 case UV_CHECK:
261 case UV_IDLE:
262 case UV_ASYNC:
263 case UV_TIMER:
264 case UV_PROCESS:
265 case UV_FS_EVENT:
266 case UV_FS_POLL:
267 case UV_POLL:
268 break;
269
270 case UV_SIGNAL:
271 /* If there are any caught signals "trapped" in the signal pipe,
272 * we can't call the close callback yet. Reinserting the handle
273 * into the closing queue makes the event loop spin but that's
274 * okay because we only need to deliver the pending events.
275 */
276 sh = (uv_signal_t*) handle;
277 if (sh->caught_signals > sh->dispatched_signals) {
278 handle->flags ^= UV_HANDLE_CLOSED;
279 uv__make_close_pending(handle); /* Back into the queue. */
280 return;
{{281 }}}
282 break;
283
284 case UV_NAMED_PIPE:
285 case UV_TCP:
286 case UV_TTY:
287 uv__stream_destroy((uv_stream_t*)handle);
288 break;
289
290 case UV_UDP:
291 uv__udp_finish_close((uv_udp_t*)handle);
292 break;
293
294 default:
295 assert(0);
296 break;
{{297 }}}
298
299 uv__handle_unref(handle);
300 QUEUE_REMOVE(&handle->handle_queue);
301
302 if (handle->close_cb) {
303 handle->close_cb(handle);
{{304 }}}
305}
Here's the asan report
10 ==13358==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d0002ec048 at pc 0x0008006a7f84 bp 0x7fffffffe8c0 sp 0x7fffffffe8b8
11 WRITE of size 8 at 0x61d0002ec048 thread T0
12 #0 0x8006a7f83 in uv__finish_close /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:300:3
13 #1 0x8006a38db in uv__run_closing_handles /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:317:5
14 #2 0x8006a3463 in uv_run /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:387:5
15 #3 0x800b378b5 in leader_lead_lh /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1039:5
16 #4 0x800b37a5c in pn_proactor_wait /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1072:15
17 #5 0x8005b55c0 in thread_run /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1051:36
18 #6 0x8005b524b in qd_server_run /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1453:5
19 #7 0x2b8e20 in main_process /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/router/src/main.c:130:5
20 #8 0x2b7ef2 in main /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/router/src/main.c:384:9
21 #9 0x2b780a in _start /x/eng/bbrtp2/daemon/DOT/dev/daemon_base/bsd99/bedrock/private/x86_64/bsd_libcrt_files/lib/csu/amd64/../../../../../../../src/lib/csu/amd64/crt1.c:74:7
22 #10 0x8002e4fff (<unknown module>)
{{23 }}
24 0x61d0002ec048 is located 1992 bytes inside of 2408-byte region [0x61d0002eb880,0x61d0002ec1e8)
25 freed by thread T5 here:
26 #0 0x28ca2d in free /x/eng/bbrtp2/daemon/DOT/dev/daemon_base/third_party/open_source/llvm/distro/compiler-rt/lib/asan/asan_malloc_linux.cpp:123:3
27 #1 0x800b3aec7 in pconnection_free /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:363:3
28 #2 0x800b3ae05 in on_close_pconnection_final /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:421:3
29 #3 0x8006a7e28 in uv__finish_close /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:303:5
30 #4 0x8006a38db in uv__run_closing_handles /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:317:5
31 #5 0x8006a3463 in uv_run /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/libuv/bedrock/internal/x86_64.asan.sim/build_configure/../../../../distro/src/unix/core.c:387:5
32 #6 0x800b378b5 in leader_lead_lh /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1039:5
33 #7 0x800b37a5c in pn_proactor_wait /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1072:15
34 #8 0x8005b55c0 in thread_run /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1051:36
35 #9 0x8005be392 in _thread_init /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/posix/threading.c:172:12
36 #10 0x800eb20f4 in thread_start /x/eng/bbrtp2/daemon/DOT/dev/daemon_base/bsd99/bedrock/private/x86_64/bsd_libthr/lib/libthr/../../../../../../src/lib/libthr/thread/thr_create.c:300:16
{{37 }}
38 previously allocated by thread T5 here:
39 #0 0x28cd12 in calloc /x/eng/bbrtp2/daemon/DOT/dev/daemon_base/third_party/open_source/llvm/distro/compiler-rt/lib/asan/asan_malloc_linux.cpp:154:3
40 #1 0x800b38288 in pconnection /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:339:39
41 #2 0x800b381e0 in pn_proactor_connect2 /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/libuv.c:1168:23
42 #3 0x800b4658d in pn_proactor_connect /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-proton/distro/c/src/proactor/proactor-internal.c:97:3
43 #4 0x8005b8156 in try_open_lh /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1169:5
44 #5 0x8005b641e in try_open_cb /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/server.c:1278:14
45 #6 0x8005bf6ad in qd_timer_visit /x/eng/bbrtp3/users/alward/resilasan_5930910_2102181129/third_party/open_source/qpid-dispatch/distro/src/timer.c:201:9
46 #7 0x8005b6ef8 (/usr/lib/libqpid-dispatch.so (distro/router/src/main.c:43)