Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9767

ASAN crash during coordinator runtime filter updates

    XMLWordPrintableJSON

    Details

    • Epic Color:
      ghx-label-13

      Description

      ASAN crash output:

      Error MessageAddress Sanitizer message detected in /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/ee_tests/impalad.ERRORStandard Error==4808==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f6288cbe818 at pc 0x00000199f6fe bp 0x7f63c1a8b270 sp 0x7f63c1a8aa20
      READ of size 1048576 at 0x7f6288cbe818 thread T73 (rpc reactor-552)
          #0 0x199f6fd in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned long, unsigned long) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
          #1 0x19a1f57 in read_msghdr(void*, __sanitizer::__sanitizer_msghdr*, long) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2781
          #2 0x19a46c3 in __interceptor_sendmsg /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2796
          #3 0x372034d in kudu::Socket::Writev(iovec const*, int, long*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/net/socket.cc:447:3
          #4 0x331c095 in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/transfer.cc:227:26
          #5 0x3324da1 in kudu::rpc::Connection::WriteHandler(ev::io&, int) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/connection.cc:802:31
          #6 0x52ca4e2 in ev_invoke_pending (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x52ca4e2)
          #7 0x32aeadc in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:196:3
          #8 0x52cdb03 in ev_run (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x52cdb03)
          #9 0x32aecd1 in kudu::rpc::ReactorThread::RunThread() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:497:9
          #10 0x32c08db in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
          #11 0x2148c26 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
          #12 0x2144b29 in kudu::Thread::SuperviseThread(void*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.cc:675:3
          #13 0x7f6c0bcf4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
          #14 0x7f6c0885834c in __clone (/lib64/libc.so.6+0xf834c)
      
      0x7f6288cbe818 is located 24 bytes inside of 1052640-byte region [0x7f6288cbe800,0x7f6288dbf7e0)
      freed by thread T114 here:
          #0 0x1a773e0 in operator delete(void*) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_new_delete.cc:137
          #1 0x7f6c090faed3 in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:110
          #2 0x7f6c090faed3 in std::string::_Rep::_M_destroy(std::allocator<char> const&) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:449
          #3 0x7f6c090faed3 in std::string::_Rep::_M_dispose(std::allocator<char> const&) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:249
          #4 0x7f6c090faed3 in std::string::reserve(unsigned long) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:511
          #5 0x2781865 in impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/client-request-state.cc:1451:11
          #6 0x26d57d5 in impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*, impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impala-server.cc:2694:19
          #7 0x266bd65 in impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*, impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
          #8 0x27a1eed in std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::DataStreamServiceIf::DataStreamServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_5>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039:2
          #9 0x3312e70 in std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439:14
          #10 0x3312231 in kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
          #11 0x22f557b in impala::ImpalaServicePool::RunThread() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15
          #12 0x22fe3fb in boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
          #13 0x2148c26 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
          #14 0x2a8c116 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:360:3
          #15 0x2a97998 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
          #16 0x2a977eb in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
          #17 0x41a8939 in thread_proxy (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a8939)
      
      previously allocated by thread T109 here:
          #0 0x1a76668 in operator new(unsigned long) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
          #1 0x7f6c090fa168 in __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:104
          #2 0x7f6c090fa168 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:607
          #3 0x7f6c090fbb7b in _S_construct_aux<char const*> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:1743
          #4 0x7f6c090fbb7b in _S_construct<char const*> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:1764
          #5 0x7f6c090fbb7b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:207
          #6 0x338b610 in impala::Coordinator::FilterState::ApplyUpdate(impala::UpdateFilterParamsPB const&, impala::Coordinator*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1417:51
          #7 0x338a4c0 in impala::Coordinator::UpdateFilter(impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1315:12
          #8 0x2781865 in impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/client-request-state.cc:1451:11
          #9 0x26d57d5 in impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*, impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impala-server.cc:2694:19
          #10 0x266bd65 in impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*, impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
          #11 0x27a1eed in std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::DataStreamServiceIf::DataStreamServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_5>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039:2
          #12 0x3312e70 in std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439:14
          #13 0x3312231 in kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
          #14 0x22f557b in impala::ImpalaServicePool::RunThread() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15
          #15 0x22fe3fb in boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
          #16 0x2148c26 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
          #17 0x2a8c116 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:360:3
          #18 0x2a97998 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
          #19 0x2a977eb in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
          #20 0x41a8939 in thread_proxy (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a8939)
      
      Thread T73 (rpc reactor-552) created by T0 here:
          #0 0x198755d in __interceptor_pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
          #1 0x2143d2c in kudu::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.cc:619:15
          #2 0x32b8755 in kudu::Status kudu::Thread::Create<void (kudu::rpc::ReactorThread::*)(), kudu::rpc::ReactorThread*>(std::string const&, std::string const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.h:164:12
          #3 0x32ae5a4 in kudu::rpc::ReactorThread::Init() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:188:10
          #4 0x32b6602 in kudu::rpc::Reactor::Init() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:762:18
          #5 0x329bffb in kudu::rpc::Messenger::Init() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:447:5
          #6 0x329b6ab in kudu::rpc::MessengerBuilder::Build(std::shared_ptr<kudu::rpc::Messenger>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:203:3
          #7 0x22dae9f in impala::RpcMgr::Init(impala::TNetworkAddress const&) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:151:3
          #8 0x2343545 in impala::ExecEnv::Init() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/exec-env.cc:385:3
          #9 0x26a8422 in ImpaladMain(int, char**) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impalad-main.cc:73:3
          #10 0x1a7aa9d in main /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
          #11 0x7f6c08781c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
      
      Thread T114 created by T0 here:
          #0 0x198755d in __interceptor_pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
          #1 0x41a7cc9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a7cc9)
          #2 0x45e0360d  (<unknown module>)
      
      Thread T109 created by T0 here:
          #0 0x198755d in __interceptor_pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
          #1 0x41a7cc9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a7cc9)
          #2 0x45e0360d  (<unknown module>)
      
      SUMMARY: AddressSanitizer: heap-use-after-free /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904 in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned long, unsigned long)
      Shadow bytes around the buggy address:
        0x0fecd118fcb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0fecd118fcc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0fecd118fcd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0fecd118fce0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0fecd118fcf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      =>0x0fecd118fd00: fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd
        0x0fecd118fd10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0fecd118fd20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0fecd118fd30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0fecd118fd40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0fecd118fd50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07 
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      ==4808==ABORTING 

        Attachments

        1. consoleFull_asan_939.txt
          3.10 MB
          Fang-Yu Rao

          Issue Links

            Activity

              People

              • Assignee:
                rizaon Riza Suminto
                Reporter:
                stakiar Sahil Takiar
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: