Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5245

buffer-allocator-test failed in ASAN build

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None

      Description

      Looks like certain large allocation in sub may have caused ASAN to allocate a huge amount of memory and failed. Tim Armstrong, can you please take a first look and reassign if necessary ?

      00:00:30 [ RUN      ] SystemAllocatorTest.LargeAllocFailure
      00:00:30 ==26508==WARNING: AddressSanitizer failed to allocate 0x1000000000000 bytes
      00:00:30 ==26508==AddressSanitizer's allocator is terminating the process instead of returning 0
      00:00:30 ==26508==If you don't like this behavior set allocator_may_return_null=1
      00:00:30 ==26508==AddressSanitizer CHECK failed: /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc:147 "((0)) != (0)" (0x0, 0x0)
      00:00:30     #0 0x102115d in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_rtl.cc:67
      00:00:30     #1 0x1027c93 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:159
      00:00:30     #2 0x10259e6 in __sanitizer::ReportAllocatorCannotReturnNull() /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc:147
      00:00:30     #3 0xf7aae6 in __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback>, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 4398046511104ul, 0ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>, __asan::AsanMapUnmapCallback> >, __sanitizer::LargeMmapAllocator<__asan::AsanMapUnmapCallback> >::ReturnNullOrDie() /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator.h:1317
      00:00:30     #4 0xf7aae6 in __asan::Allocator::Allocate(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType, bool) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_allocator.cc:359
      00:00:30     #5 0x1017821 in __interceptor_malloc /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:53
      00:00:30     #6 0x107e81f in impala::SystemAllocator::Allocate(long, impala::BufferPool::BufferHandle*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/bufferpool/system-allocator.cc:56:45
      00:00:30     #7 0x104daae in impala::SystemAllocatorTest_LargeAllocFailure_Test::TestBody() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/bufferpool/buffer-allocator-test.cc:173:19
      00:00:30     #8 0x2c0d552 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/bufferpool/buffer-allocator-test+0x2c0d552)
      00:00:30     #9 0x2c040c9 in testing::Test::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/bufferpool/buffer-allocator-test+0x2c040c9)
      00:00:30     #10 0x2c04217 in testing::TestInfo::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/bufferpool/buffer-allocator-test+0x2c04217)
      00:00:30     #11 0x2c042f4 in testing::TestCase::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/bufferpool/buffer-allocator-test+0x2c042f4)
      00:00:30     #12 0x2c05577 in testing::internal::UnitTestImpl::RunAllTests() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/bufferpool/buffer-allocator-test+0x2c05577)
      00:00:30     #13 0x2c05852 in testing::UnitTest::Run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/bufferpool/buffer-allocator-test+0x2c05852)
      00:00:30     #14 0x104e18f in main /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/bufferpool/buffer-allocator-test.cc:192:12
      00:00:30     #15 0x3be821ecdc in __libc_start_main (/lib64/libc.so.6+0x3be821ecdc)
      00:00:30     #16 0xf71c54 in _start (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/runtime/bufferpool/buffer-allocator-test+0xf71c54)
      00:00:30 
      00:00:30 
      

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5245: fix ASAN buffer-allocator-test

        • Use the allocator_may_return_null=1 ASAN option so that the allocation
          just fails instead of crashing the process.
        • Work around an bug in the implementation of the option where
          posix_memalign() does not return ENOMEM. I filed an upstream bug
          https://bugs.llvm.org/show_bug.cgi?id=32968.

        Change-Id: I2f295f841b8b6e7996d54bdf2dfc0a092dcb864c
        Reviewed-on: http://gerrit.cloudera.org:8080/6819
        Reviewed-by: Jim Apple <jbapple-impala@apache.org>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5245 : fix ASAN buffer-allocator-test Use the allocator_may_return_null=1 ASAN option so that the allocation just fails instead of crashing the process. Work around an bug in the implementation of the option where posix_memalign() does not return ENOMEM. I filed an upstream bug https://bugs.llvm.org/show_bug.cgi?id=32968 . Change-Id: I2f295f841b8b6e7996d54bdf2dfc0a092dcb864c Reviewed-on: http://gerrit.cloudera.org:8080/6819 Reviewed-by: Jim Apple <jbapple-impala@apache.org> Tested-by: Impala Public Jenkins
        Hide
        dhecht Dan Hecht added a comment -

        Disabled the test under ASAN, so no longer a broken build blocker.

        commit 5294bb2d7a7733b4c5b3fae7a14498d85e87c90d
        Author: Dan Hecht <dhecht@cloudera.com>
        Date:   Wed Apr 26 14:59:32 2017 -0700
        
            IMPALA-5245: Disable buffer-allocator-test under ASAN
        
            Until we figure out the root cause, disable this test (which exercises
            code that is under development and not yet enabled by default).
        
            Change-Id: I4e685d4482d548cc3e724c20a83ae14890ce56ec
            Reviewed-on: http://gerrit.cloudera.org:8080/6737
            Reviewed-by: Alex Behm <alex.behm@cloudera.com>
            Tested-by: Impala Public Jenkins
        
        Show
        dhecht Dan Hecht added a comment - Disabled the test under ASAN, so no longer a broken build blocker. commit 5294bb2d7a7733b4c5b3fae7a14498d85e87c90d Author: Dan Hecht <dhecht@cloudera.com> Date: Wed Apr 26 14:59:32 2017 -0700 IMPALA-5245: Disable buffer-allocator-test under ASAN Until we figure out the root cause, disable this test (which exercises code that is under development and not yet enabled by default ). Change-Id: I4e685d4482d548cc3e724c20a83ae14890ce56ec Reviewed-on: http: //gerrit.cloudera.org:8080/6737 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        kwho Michael Ho added a comment -

        As discussed offline, IMPALA-5246 was filed last week for tracking the crash in test_mem_limit.

        Show
        kwho Michael Ho added a comment - As discussed offline, IMPALA-5246 was filed last week for tracking the crash in test_mem_limit.
        Hide
        alex.behm Alexander Behm added a comment -

        Dan Hecht, don't know. Logs seem to have no ASAN output. No stack or core. I asked Michael Ho to further triage.

        Show
        alex.behm Alexander Behm added a comment - Dan Hecht , don't know. Logs seem to have no ASAN output. No stack or core. I asked Michael Ho to further triage.
        Hide
        dhecht Dan Hecht added a comment -

        Alexander Behm same crash or different? Doesn't seem like it could be related, but not 100% sure.

        Show
        dhecht Dan Hecht added a comment - Alexander Behm same crash or different? Doesn't seem like it could be related, but not 100% sure.
        Hide
        alex.behm Alexander Behm added a comment -

        Dan Hecht, ASAN is also crashing in TestUdfExecution::test_mem_limit. Not sure if it's related.

        Show
        alex.behm Alexander Behm added a comment - Dan Hecht , ASAN is also crashing in TestUdfExecution::test_mem_limit. Not sure if it's related.
        Hide
        dhecht Dan Hecht added a comment -

        I'll disable the test later today.

        Show
        dhecht Dan Hecht added a comment - I'll disable the test later today.
        Hide
        alex.behm Alexander Behm added a comment -

        Dan Hecht, can you find someone else to work on this? Tim is on PTO and our ASAN builds have been failing for over a week now.

        Show
        alex.behm Alexander Behm added a comment - Dan Hecht , can you find someone else to work on this? Tim is on PTO and our ASAN builds have been failing for over a week now.
        Hide
        tarmstrong Tim Armstrong added a comment -

        Ah should have run that test under ASAN before committing. I won't have time to work on the fix - travelling for a couple of weeks. Feel free to disable that test under ASAN or other - I can investigate when I have a chance.

        Show
        tarmstrong Tim Armstrong added a comment - Ah should have run that test under ASAN before committing. I won't have time to work on the fix - travelling for a couple of weeks. Feel free to disable that test under ASAN or other - I can investigate when I have a chance.

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            kwho Michael Ho
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development