Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4542

Use-after-free in various backend tests

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:

      Description

      Henry Robinson, is this related to commit https://github.com/apache/incubator-impala/commit/707f71b6ea13487c707337785e785487d2f470f2 or is it some sort of latent bug ? Didn't look too much into why the destructor of test_env_ was invoked twice. I thought it was single threaded.

      01:25:03 ==8313==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000047fd8 at pc 0x00000155e49e bp 0x7fffb8282ea0 sp 0x7fffb8282e98
      01:25:03 READ of size 8 at 0x611000047fd8 thread T0
      01:25:03     #0 0x155e49d in boost::scoped_ptr<impala::ThreadResourceMgr>::get() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:105:16
      01:25:03     #1 0x154f4bf in impala::RuntimeState::ReleaseResources() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/runtime-state.cc:314:5
      01:25:03     #2 0x1534de8 in impala::TestEnv::TearDownRuntimeStates() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:94:47
      01:25:03     #3 0x1534c42 in impala::TestEnv::~TestEnv() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:61:3
      01:25:03     #4 0x1045d35 in void boost::checked_delete<impala::TestEnv>(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34:5
      01:25:03     #5 0x104c7e2 in boost::scoped_ptr<impala::TestEnv>::reset(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      01:25:03     #6 0x2b76792 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0x2b76792)
      01:25:03     #7 0x2b6de38 in testing::Test::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0x2b6de38)
      01:25:03     #8 0x2b6dfb7 in testing::TestInfo::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0x2b6dfb7)
      01:25:03     #9 0x2b6e094 in testing::TestCase::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0x2b6e094)
      01:25:03     #10 0x2b6f317 in testing::internal::UnitTestImpl::RunAllTests() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0x2b6f317)
      01:25:03     #11 0x2b6f5f2 in testing::UnitTest::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0x2b6f5f2)
      01:25:03     #12 0x1032b45 in main /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/buffered-tuple-stream-test.cc:1261:11
      01:25:03     #13 0x32b4e1ecdc in __libc_start_main (/lib64/libc.so.6+0x32b4e1ecdc)
      01:25:03     #14 0xf47634 in _start (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0xf47634)
      01:25:03 
      01:25:03 0x611000047fd8 is located 88 bytes inside of 200-byte region [0x611000047f80,0x611000048048)
      01:25:03 freed by thread T0 here:
      01:25:03     #0 0x101e960 in operator delete(void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:94
      01:25:03     #1 0x1535c12 in boost::scoped_ptr<impala::ExecEnv>::reset(impala::ExecEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      01:25:03     #2 0x1534c4c in impala::TestEnv::~TestEnv() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:62:3
      01:25:03     #3 0x1045d35 in void boost::checked_delete<impala::TestEnv>(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34:5
      01:25:03     #4 0x104c7e2 in boost::scoped_ptr<impala::TestEnv>::reset(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      01:25:03     #5 0x2b76792 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0x2b76792)
      01:25:03 
      01:25:03 previously allocated by thread T0 here:
      01:25:03     #0 0x101e360 in operator new(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:62
      01:25:03     #1 0x15346bc in impala::TestEnv::TestEnv() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:39:19
      01:25:03     #2 0x104194e in impala::SimpleTupleStreamTest::SetUp() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/buffered-tuple-stream-test.cc:71:25
      01:25:03     #3 0x2b76792 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-tuple-stream-test+0x2b76792)
      01:25:03 
      
      01:25:03 ==7120==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000047c18 at pc 0x00000155fc3e bp 0x7fff66257120 sp 0x7fff66257118
      01:25:03 READ of size 8 at 0x611000047c18 thread T0
      01:25:03     #0 0x155fc3d in boost::scoped_ptr<impala::ThreadResourceMgr>::get() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:105:16
      01:25:03     #1 0x1550caf in impala::RuntimeState::ReleaseResources() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/runtime-state.cc:314:5
      01:25:03     #2 0x1541578 in impala::TestEnv::TearDownRuntimeStates() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:94:47
      01:25:03     #3 0x1034143 in impala::BufferedBlockMgrTest_GetNewBlockSmallBlocks_Test::TestBody() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/buffered-block-mgr-test.cc:648:3
      01:25:03     #4 0x2b94ff2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0x2b94ff2)
      01:25:03     #5 0x2b8c949 in testing::Test::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0x2b8c949)
      01:25:03     #6 0x2b8ca97 in testing::TestInfo::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0x2b8ca97)
      01:25:03     #7 0x2b8cb74 in testing::TestCase::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0x2b8cb74)
      01:25:03     #8 0x2b8ddf7 in testing::internal::UnitTestImpl::RunAllTests() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0x2b8ddf7)
      01:25:03     #9 0x2b8e0d2 in testing::UnitTest::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0x2b8e0d2)
      01:25:03     #10 0x1048c95 in main /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/buffered-block-mgr-test.cc:1470:11
      01:25:03     #11 0x32b4e1ecdc in __libc_start_main (/lib64/libc.so.6+0x32b4e1ecdc)
      01:25:03     #12 0xf59584 in _start (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0xf59584)
      01:25:03 
      01:25:03 0x611000047c18 is located 88 bytes inside of 200-byte region [0x611000047bc0,0x611000047c88)
      01:25:03 freed by thread T0 here:
      01:25:03     #0 0x10308b0 in operator delete(void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:94
      01:25:03     #1 0x15423a2 in boost::scoped_ptr<impala::ExecEnv>::reset(impala::ExecEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      01:25:03     #2 0x15413dc in impala::TestEnv::~TestEnv() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:62:3
      01:25:03     #3 0x10549e5 in void boost::checked_delete<impala::TestEnv>(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34:5
      01:25:03     #4 0x1073222 in boost::scoped_ptr<impala::TestEnv>::reset(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      01:25:03     #5 0x1053b2e in impala::BufferedBlockMgrTest::TearDown() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/buffered-block-mgr-test.cc:80:5
      01:25:03     #6 0x2b94ff2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0x2b94ff2)
      01:25:03 
      01:25:03 previously allocated by thread T0 here:
      01:25:03     #0 0x10302b0 in operator new(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:62
      01:25:03     #1 0x1540e4c in impala::TestEnv::TestEnv() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:39:19
      01:25:03     #2 0x1053a4e in impala::BufferedBlockMgrTest::SetUp() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/buffered-block-mgr-test.cc:75:25
      01:25:03     #3 0x2b94ff2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/buffered-block-mgr-test+0x2b94ff2)
      01:25:03 
      
      01:25:03 ==8563==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100002d098 at pc 0x000001511a6e bp 0x7fffaa1c7760 sp 0x7fffaa1c7758
      01:25:03 READ of size 8 at 0x61100002d098 thread T0
      01:25:03     #0 0x1511a6d in boost::scoped_ptr<impala::ThreadResourceMgr>::get() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:105:16
      01:25:03     #1 0x15008bf in impala::RuntimeState::ReleaseResources() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/runtime-state.cc:314:5
      01:25:03     #2 0x14e5488 in impala::TestEnv::TearDownRuntimeStates() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:94:47
      01:25:03     #3 0x14e52e2 in impala::TestEnv::~TestEnv() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:61:3
      01:25:03     #4 0x102b7b5 in void boost::checked_delete<impala::TestEnv>(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34:5
      01:25:03     #5 0x103ce92 in boost::scoped_ptr<impala::TestEnv>::reset(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      01:25:03     #6 0x1029dfb in impala::HashTableTest::TearDown() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hash-table-test.cc:87:5
      01:25:03     #7 0x2b622f2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0x2b622f2)
      01:25:03     #8 0x2b59c18 in testing::Test::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0x2b59c18)
      01:25:03     #9 0x2b59d97 in testing::TestInfo::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0x2b59d97)
      01:25:03     #10 0x2b59e74 in testing::TestCase::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0x2b59e74)
      01:25:03     #11 0x2b5b0f7 in testing::internal::UnitTestImpl::RunAllTests() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0x2b5b0f7)
      01:25:03     #12 0x2b5b3d2 in testing::UnitTest::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0x2b5b3d2)
      01:25:03     #13 0x101e04e in main /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hash-table-test.cc:636:11
      01:25:03     #14 0x32b4e1ecdc in __libc_start_main (/lib64/libc.so.6+0x32b4e1ecdc)
      01:25:03     #15 0xf43714 in _start (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0xf43714)
      01:25:03 
      01:25:03 0x61100002d098 is located 88 bytes inside of 200-byte region [0x61100002d040,0x61100002d108)
      01:25:03 freed by thread T0 here:
      01:25:03     #0 0x101aa40 in operator delete(void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:94
      01:25:03     #1 0x14e62b2 in boost::scoped_ptr<impala::ExecEnv>::reset(impala::ExecEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      01:25:03     #2 0x14e52ec in impala::TestEnv::~TestEnv() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:62:3
      01:25:03     #3 0x102b7b5 in void boost::checked_delete<impala::TestEnv>(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34:5
      01:25:03     #4 0x103ce92 in boost::scoped_ptr<impala::TestEnv>::reset(impala::TestEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      01:25:03     #5 0x1029dfb in impala::HashTableTest::TearDown() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hash-table-test.cc:87:5
      01:25:03     #6 0x2b622f2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0x2b622f2)
      01:25:03 
      01:25:03 previously allocated by thread T0 here:
      01:25:03     #0 0x101a440 in operator new(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:62
      01:25:03     #1 0x14e4d5c in impala::TestEnv::TestEnv() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/test-env.cc:39:19
      01:25:03     #2 0x1029122 in impala::HashTableTest::SetUp() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hash-table-test.cc:64:25
      01:25:03     #3 0x2b622f2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exec/hash-table-test+0x2b622f2)
      

        Activity

        Hide
        henryr Henry Robinson added a comment -

        The problem I think is that RuntimeState::ReleaseResources() accesses ExecEnv::GetInstance(); however the TestEnv d'tor deletes the singleton instance so the next test will hit use-after-free.

        Fixing this problem is easy, since RuntimeState actually has its own ExecEnv pointer. But maybe a better approach is a testing-only 'install new exec env' call that replaces the singleton. I know that there's an upcoming change where we remove RuntimeState::exec_env_ completely, perhaps we can defer that change until then.

        Show
        henryr Henry Robinson added a comment - The problem I think is that RuntimeState::ReleaseResources() accesses ExecEnv::GetInstance() ; however the TestEnv d'tor deletes the singleton instance so the next test will hit use-after-free. Fixing this problem is easy, since RuntimeState actually has its own ExecEnv pointer. But maybe a better approach is a testing-only 'install new exec env' call that replaces the singleton. I know that there's an upcoming change where we remove RuntimeState::exec_env_ completely, perhaps we can defer that change until then.
        Show
        henryr Henry Robinson added a comment - Fixed in https://github.com/apache/incubator-impala/commit/8adc49e51c44aa3a4b5334a93b64ae5a21e400b9

          People

          • Assignee:
            henryr Henry Robinson
            Reporter:
            kwho Michael Ho
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development