Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4205

tmp-file-mgr-test: AddressSanitizer: heap-use-after-free

    Details

      Description

      Tim Armstrong, I'm assigning this to you, as I think the culprit commit is:

      IMPALA-3671: Add query option to limit scratch space usage
      
      $ be/build/debug/runtime/tmp-file-mgr-test
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/home/mikeb/Impala/fe/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/home/mikeb/Impala/testdata/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      [==========] Running 6 tests from 1 test case.
      [----------] Global test environment set-up.
      [----------] 6 tests from TmpFileMgrTest
      [ RUN      ] TmpFileMgrTest.TestFileAllocation
      =================================================================
      ==17776==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000056bb8 at pc 0x000000f5e78b bp 0x7fffb9d00dd0 sp 0x7fffb9d00580
      READ of size 1191182448 at 0x60b000056bb8 thread T0
          #0 0xf5e78a in memcpy /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-14-04/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/a
      san/asan_interceptors.cc:438
          #1 0x7f1680a8ac7f in std::char_traits<char>::copy(char*, char const*, unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-14-04/too
      lchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/char_traits.h:275
          #2 0x7f1680a8ac7f in std::string::_M_copy(char*, char const*, unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-14-04/toolchain/s
      ource/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:361
          #3 0x7f1680a8ac7f in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-1
      4-04/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:631
          #4 0x7f1680a8b3eb in std::string::_Rep::_M_grab(std::allocator<char> const&, std::allocator<char> const&) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-pac
      kage-ubuntu-14-04/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:229
          #5 0x7f1680a8b3eb in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) /data/jenkins/workspace/verify-impala-toolchain-package
      -build/label/ec2-package-ubuntu-14-04/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:173
          #6 0xfdf4b2 in impala::TmpFileMgrTest_TestFileAllocation_Test::TestBody() /home/mikeb/Impala/be/src/runtime/tmp-file-mgr-test.cc:104:135
          #7 0x2aa2dd2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/mikeb/Impala/be/build
      /debug/runtime/tmp-file-mgr-test+0x2aa2dd2)
          #8 0x2a99949 in testing::Test::Run() (/home/mikeb/Impala/be/build/debug/runtime/tmp-file-mgr-test+0x2a99949)
          #9 0x2a99a97 in testing::TestInfo::Run() (/home/mikeb/Impala/be/build/debug/runtime/tmp-file-mgr-test+0x2a99a97)
          #10 0x2a99b74 in testing::TestCase::Run() (/home/mikeb/Impala/be/build/debug/runtime/tmp-file-mgr-test+0x2a99b74)
          #11 0x2a9adf7 in testing::internal::UnitTestImpl::RunAllTests() (/home/mikeb/Impala/be/build/debug/runtime/tmp-file-mgr-test+0x2a9adf7)
          #12 0x2a9b0d2 in testing::UnitTest::Run() (/home/mikeb/Impala/be/build/debug/runtime/tmp-file-mgr-test+0x2a9b0d2)
          #13 0xfe66ae in main /home/mikeb/Impala/be/src/runtime/tmp-file-mgr-test.cc:279:11
          #14 0x7f167feeef44 in __libc_start_main /build/eglibc-oGUzwX/eglibc-2.19/csu/libc-start.c:287
          #15 0xf04d51 in _start (/home/mikeb/Impala/be/build/debug/runtime/tmp-file-mgr-test+0xf04d51)
      
      0x60b000056c05 is located 0 bytes to the right of 101-byte region [0x60b000056ba0,0x60b000056c05)
      freed by thread T0 here:
          #0 0xfdc050 in operator delete(void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-14-04/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:94
          #1 0xff15f5 in std::default_delete<impala::TmpFileMgr::File>::operator()(impala::TmpFileMgr::File*) const /home/mikeb/Impala/toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/unique_ptr.h:76:2
          #2 0xff1590 in std::unique_ptr<impala::TmpFileMgr::File, std::default_delete<impala::TmpFileMgr::File> >::~unique_ptr() /home/mikeb/Impala/toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/unique_ptr.h:236:4
          #3 0xff151f in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<impala::TmpFileMgr::File, std::default_delete<impala::TmpFileMgr::File> >*>(std::unique_ptr<impala::TmpFileMgr::File, std::default_delete<impala::TmpFileMgr::File> >*, std::unique_ptr<impala::TmpFileMgr::File, std::default_delete<impala::TmpFileMgr::File> >*) /home/mikeb/Impala/toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_construct.h:103:6
          #4 0x108b108 in std::vector<std::unique_ptr<impala::TmpFileMgr::File, std::default_delete<impala::TmpFileMgr::File> >, std::allocator<std::unique_ptr<impala::TmpFileMgr::File, std::default_delete<impala::TmpFileMgr::File> > > >::_M_erase_at_end(std::unique_ptr<impala::TmpFileMgr::File, std::default_delete<impala::TmpFileMgr::File> >*) /home/mikeb/Impala/toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_vector.h:1438:2
          #5 0x107c404 in impala::TmpFileMgr::FileGroup::Close() /home/mikeb/Impala/be/src/runtime/tmp-file-mgr.cc:293:3
          #6 0xfdf48a in impala::TmpFileMgrTest_TestFileAllocation_Test::TestBody() /home/mikeb/Impala/be/src/runtime/tmp-file-mgr-test.cc:103:3
          #7 0x2aa2dd2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/mikeb/Impala/be/build/debug/runtime/tmp-file-mgr-test+0x2aa2dd2)
      
      previously allocated by thread T0 here:
          #0 0xfdba50 in operator new(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-14-04/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:62
          #1 0x7f1680a89fe8 in __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-14-04/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:104
          #2 0x7f1680a89fe8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-14-04/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:607
      
      SUMMARY: AddressSanitizer: heap-use-after-free /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-14-04/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:438 in memcpy
      Shadow bytes around the buggy address:
        0x0c1680002d20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c1680002d30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c1680002d40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c1680002d50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c1680002d60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      =>0x0c1680002d70: fa fa fa fa fd fd fd[fd]fd fd fd fd fd fd fd fd
        0x0c1680002d80: fd fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
        0x0c1680002d90: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
        0x0c1680002da0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa
        0x0c1680002db0: fa fa fa fa fa fa 00 00 00 00 00 00 00 00 00 00
        0x0c1680002dc0: 00 00 00 05 fa fa fa fa fa fa fa fa 00 00 00 00
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07
        Heap left redzone:       fa
        Heap right redzone:      fb
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack partial redzone:   f4
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      ==17776==ABORTING
      $ 
      

      http://sandbox.jenkins.cloudera.com/view/Impala/view/Evergreen-asf-master/job/impala-asf-master-core-asan/86/

      I guess because ASAN forces an abort, the test XML isn't written. But I can reproduce this in my dev environment.

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-4205: fix tmp-file-mgr-test under ASAN

        Fix use-after-free of 'file' object.

        Change-Id: Ife70225924e6da76fb835ff7b140cb71c87d411d
        Reviewed-on: http://gerrit.cloudera.org:8080/4546
        Reviewed-by: Dan Hecht <dhecht@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-4205 : fix tmp-file-mgr-test under ASAN Fix use-after-free of 'file' object. Change-Id: Ife70225924e6da76fb835ff7b140cb71c87d411d Reviewed-on: http://gerrit.cloudera.org:8080/4546 Reviewed-by: Dan Hecht <dhecht@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            mikesbrown Michael Brown
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development