Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5172

crash in tcmalloc::CentralFreeList::FetchFromOneSpans

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:

      Description

      This was encountered during an automated test run.

      #0  0x0000003556e328e5 in raise () from /lib64/libc.so.6
      #1  0x0000003556e340c5 in abort () from /lib64/libc.so.6
      #2  0x00007fea49b64c55 in os::abort(bool) () from /opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #3  0x00007fea49ce6cd7 in VMError::report_and_die() () from /opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #4  0x00007fea49b69b6f in JVM_handle_linux_signal () from /opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #5  <signal handler called>
      #6  0x0000000001bf2253 in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) ()
      #7  0x0000000001bf254c in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
      #8  0x0000000001bf25f4 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
      #9  0x0000000001bffcc3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) ()
      #10 0x0000000001c0eea8 in tc_newarray ()
      #11 0x0000000000bcbd24 in allocate (this=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/ext/new_allocator.h:104
      #12 allocate (this=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/alloc_traits.h:357
      #13 _M_allocate (this=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_vector.h:170
      #14 std::vector<impala::TRuntimeProfileNode, std::allocator<impala::TRuntimeProfileNode> >::_M_emplace_back_aux<impala::TRuntimeProfileNode> (this=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/vector.tcc:412
      #15 0x0000000000bc5747 in emplace_back<impala::TRuntimeProfileNode> (this=0x93c3400, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/vector.tcc:101
      #16 push_back (this=0x93c3400, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_vector.h:932
      #17 impala::RuntimeProfile::ToThrift (this=0x93c3400, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:758
      #18 0x0000000000bc54b2 in impala::RuntimeProfile::ToThrift (this=0x29480a00, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:836
      #19 0x0000000000bc54b2 in impala::RuntimeProfile::ToThrift (this=0x1ebf6300, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:836
      #20 0x0000000000bc54b2 in impala::RuntimeProfile::ToThrift (this=0x78d1700, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:836
      #21 0x0000000000bc54b2 in impala::RuntimeProfile::ToThrift (this=0x598f700, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:836
      #22 0x0000000000bc54b2 in impala::RuntimeProfile::ToThrift (this=0xaa04f00, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:836
      #23 0x0000000000bc54b2 in impala::RuntimeProfile::ToThrift (this=0x1fdd0460, nodes=0x7fe9ee3f6d78) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:836
      #24 0x0000000000bc5c1d in impala::RuntimeProfile::SerializeToArchiveString (this=) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:724
      #25 0x0000000000bc632f in impala::RuntimeProfile::SerializeToArchiveString (this=0x1fdd0460) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile.cc:718
      #26 0x0000000000abb6c4 in impala::ImpalaServer::ArchiveQuery (this=0x7bd9c00, query=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/impala-server.cc:685
      #27 0x0000000000abc6e6 in impala::ImpalaServer::UnregisterQuery (this=0x7bd9c00, query_id=) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/impala-server.cc:986
      #28 0x0000000000af6a42 in impala::ImpalaServer::close (this=0x7bd9c00, handle=) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/impala-beeswax-server.cc:236
      #29 0x0000000000d523f5 in beeswax::BeeswaxServiceProcessor::process_close (this=0x7279ae0, seqid=0, iprot=) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3543
      #30 0x0000000000d59c54 in beeswax::BeeswaxServiceProcessor::dispatchCall (this=) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:2952
      #31 0x000000000080f24c in apache::thrift::TDispatchProcessor::process (this=0x7279ae0, in=..., out=..., connectionContext=0xa817d80) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/thrift-0.9.0-p8/include/thrift/TDispatchProcessor.h:121
      #32 0x0000000001b4291b in apache::thrift::server::TThreadPoolServer::Task::run() ()
      #33 0x0000000001b2a4e9 in apache::thrift::concurrency::ThreadManager::Worker::run() ()
      #34 0x00000000009fdbe9 in impala::ThriftThread::RunRunnable (this=) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/rpc/thrift-thread.cc:64
      #35 0x00000000009fe642 in operator() (function_obj_ptr=) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:280
      #36 operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<long unsigned int>*>, boost::_bi::list0> (function_obj_ptr=) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:392
      #37 operator() (function_obj_ptr=) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #38 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> > >, void>::invoke (function_obj_ptr=) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:153
      #39 0x0000000000be2479 in operator() (name=) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767
      #40 impala::Thread::SuperviseThread (name=) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:325
      #41 0x0000000000be2ec4 in operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0> (this=0x7de3c00) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457
      #42 operator() (this=0x7de3c00) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #43 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x7de3c00) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116
      #44 0x0000000000e4e33a in thread_proxy ()
      #45 0x0000003557207851 in start_thread () from /lib64/libpthread.so.0
      #46 0x0000003556ee894d in clone () from /lib64/libc.so.6

        Attachments

        1. IMPALA-5172.patch
          2 kB
          Michael Ho
        2. impalad_node2.ERROR
          12 kB
          Joe McDonnell
        3. impalad_node1.ERROR
          12 kB
          Joe McDonnell

          Issue Links

            Activity

              People

              • Assignee:
                joemcdonnell Joe McDonnell
                Reporter:
                mmulder Matthew Mulder
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: