[IMPALA-12915] double-free memory issue in unifiedbetests if it's built in dynamically linked mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Backend
• Epic Color: ghx-label-3
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 18/Mar/24 07:50
• Updated: 26/Mar/24 00:36
• Resolved: 26/Mar/24 00:36

Description

I have a pretty simple change that just adds one new metric in common/thrift/metrics.json: https://github.com/stiga-huang/impala/commit/3972aed68428b3f307e823f49a2700c1a98aa3b3
It breaks the clang-tidy job in the step of running "unifiedbetests --gtest_list_tests".

The job builds Impala using the -so option which means dynamically link executables (default is static). I'm able to reproduce the issue locally. Using gdb, I can see the process crashed by a memory issue:

(gdb) bt
#0  0x0000000002ff8d6b in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int) ()
#1  0x0000000002ff93c5 in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned int) ()
#2  0x00007fffef06db1f in std::_Function_base::_Base_manager<boost::algorithm::detail::is_any_ofF<char> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) ()
   from /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
#3  0x00007fffef06d7e4 in std::_Function_handler<bool (char), boost::algorithm::detail::is_any_ofF<char> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) ()
   from /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
#4  0x000000000109fc07 in std::_Function_base::~_Function_base() ()
#5  0x00007fffed7a8605 in __cxa_finalize () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007fffeef995a3 in __do_global_dtors_aux () from /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
#7  0x00007fffffff03c0 in ?? ()

boroknagyz found that upgrading gtest to 1.14 resolves the issue. The current gtest version is 1.6. I tried building with ASAN and also build the underlying gtest-1.6 with ASAN. The run then shows the memory issue:

==1778==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000098530 at pc 0x00000123a9a9 bp 0x7ffeeaf47d60 sp 0x7ffeeaf47d58
READ of size 8 at 0x603000098530 thread T0
    #0 0x123a9a8 in testing::internal::String::~String() /home/quanlong/workspace/Impala/toolchain/toolchain-packages-gcc10.4.0/gtest-1.6.0/include/gtest/internal/gtest-string.h:218:24
    #1 0x530fc2f in void std::_Destroy_aux<false>::__destroy<testing::internal::String*>(testing::internal::String*, testing::internal::String*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_construct.h:108:6
    #2 0x52fdcdd in std::vector<testing::internal::String, std::allocator<testing::internal::String> >::~vector() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:677:2
    #3 0x7f4ca763a604 in __cxa_finalize /build/glibc-CVJwZb/glibc-2.27/stdlib/cxa_finalize.c:83
    #4 0x7f4ca93a8cb2 in __do_global_dtors_aux (/home/quanlong/workspace/Impala/be/src/kudu/util/libkudu_test_util.so+0x6ecb2)

0x603000098530 is located 0 bytes inside of 32-byte region [0x603000098530,0x603000098550)
freed by thread T0 here:
    #0 0x1233dcf in operator delete(void*) (/home/quanlong/workspace/Impala/be/build/debug/service/unifiedbetests+0x1233dcf)
    #1 0x5310da3 in std::_Vector_base<testing::internal::String, std::allocator<testing::internal::String> >::~_Vector_base() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:332:2
    #2 0x7f4ca763a030 in __run_exit_handlers /build/glibc-CVJwZb/glibc-2.27/stdlib/exit.c:108

previously allocated by thread T0 here:
    #0 0x123305f in operator new(unsigned long) (/home/quanlong/workspace/Impala/be/build/debug/service/unifiedbetests+0x123305f)
    #1 0x530fdd0 in std::vector<testing::internal::String, std::allocator<testing::internal::String> >::_M_realloc_insert(__gnu_cxx::__normal_iterator<testing::internal::String*, std::vector<testing::internal::String, std::allocator<testing::internal::String> > >, testing::internal::String const&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/vector.tcc:440:33
    #2 0x52fc053 in void testing::internal::InitGoogleTestImpl<char>(int*, char**) /media/quanlong/hdd-backup/native-toolchain/source/gtest/gtest-1.6.0/src/gtest.cc:4868:13
    #3 0x1236435 in main /home/quanlong/workspace/Impala/be/src/service/unified-betest-main.cc:40:3
    #4 0x7f4ca7618c86 in __libc_start_main /build/glibc-CVJwZb/glibc-2.27/csu/../csu/libc-start.c:310

Note that I have to remove the definition of "THttpTransport::CRLF_LEN" in be/src/transport/THttpTransport.cpp to avoid an ODR violation issue.

libkudu_test_util.so wants to delete a vector that is already deleted by gtest.
https://github.com/google/googletest/blob/release-1.6.0/include/gtest/internal/gtest-port.h#L1057
 
In the static build, we are fine since there are no libkudu_test_util.so or libkudu_util.so.

Issue Links

• relates to: IMPALA-12886 Bump GoogleTest to 1.14.0
• relates to: IMPALA-12603 Run clang-tidy on ARM in commit tests
• links to: Toolchain patch