Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4569

Multi-threaded compute stats DCHECK: impala::MemPool::~MemPool

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:

      Description

      Query:

      USE tpch_100_parquet;
      SET MT_DOP=16;
      SET MEM_LIMIT=150M;
      COMPUTE STATS partsupp;
      

      There was only 1 query executing, the stress test was in "binary search" mode. This occurred on a physical 8 node cluster

      Stack Trace:

      #0  0x0000003864032625 in raise () from /lib64/libc.so.6
      #1  0x0000003864033e05 in abort () from /lib64/libc.so.6
      #2  0x0000000002820664 in google::DumpStackTraceAndExit() ()
      #3  0x0000000002819acd in google::LogMessage::Fail() ()
      #4  0x000000000281c3f6 in google::LogMessage::SendToLog() ()
      #5  0x00000000028195ed in google::LogMessage::Flush() ()
      #6  0x000000000281ce9e in google::LogMessageFatal::~LogMessageFatal() ()
      #7  0x00000000013f324a in impala::MemPool::~MemPool (this=0x13b23cc0, 
          __in_chrg=<value optimized out>)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/mem-pool.cc:69
      #8  0x0000000001574f1b in boost::checked_delete<impala::MemPool> (x=0x13b23cc0)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
      #9  0x00000000015749f5 in boost::scoped_ptr<impala::MemPool>::~scoped_ptr (
          this=0x144b2330, __in_chrg=<value optimized out>)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:82
      #10 0x00000000016dc7d2 in impala::HdfsParquetScanner::~HdfsParquetScanner (
          this=0x144b2100, __in_chrg=<value optimized out>)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/hdfs-parquet-scanner.h:325
      #11 0x00000000016dc89e in impala::HdfsParquetScanner::~HdfsParquetScanner (
          this=0x144b2100, __in_chrg=<value optimized out>)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/hdfs-parquet-scanner.h:325
      #12 0x0000000001689184 in boost::checked_delete<impala::HdfsScanner> (x=0x144b2100)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
      #13 0x00000000016880d5 in boost::scoped_ptr<impala::HdfsScanner>::~scoped_ptr (
          this=0x7f3070284cb0, __in_chrg=<value optimized out>)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:82
      #14 0x0000000001697893 in boost::scoped_ptr<impala::HdfsScanner>::reset (this=0x13bea4c0, 
          p=0x0)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:88
      #15 0x00000000016a90bb in impala::HdfsScanNodeMt::Close (this=0x13bea000, 
          state=0x1295e000)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/hdfs-scan-node-mt.cc:121
      #16 0x000000000165949c in impala::ExecNode::Close (this=0x10ccb840, state=0x1295e000)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/exec-node.cc:197
      #17 0x000000000174b901 in impala::PartitionedAggregationNode::Close (this=0x10ccb840, 
          state=0x1295e000)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:726
      #18 0x00000000019e5064 in impala::PlanFragmentExecutor::Close (this=0x128af0a0)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/plan-fragment-executor.cc:531
      #19 0x000000000153633e in impala::FragmentMgr::FragmentExecState::Exec (this=0x128aed00)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/service/fragment-exec-state.cc:61
      #20 0x000000000152da74 in impala::FragmentMgr::FragmentThread (this=0x144d4740, 
          fragment_instance_id=...)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/service/fragment-mgr.cc:86
      #21 0x00000000015317f6 in boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>::operator() (this=0x123fd920, p=0x144d4740, a1=...)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/bind/mem_fn_template.hpp:165
      #22 0x00000000015315b3 in boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> >::operator()<boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list0> (this=0x123fd930, f=..., a=...)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/bind/bind.hpp:313
      #23 0x0000000001530edd in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> > >::operator() (this=0x123fd920)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20
      #24 0x0000000001530870 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> > >, void>::invoke (function_obj_ptr=...)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/function/function_template.hpp:153
      #25 0x000000000134404c in boost::function0<void>::operator() (this=0x7f3070285c40)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/function/function_template.hpp:767
      #26 0x00000000015f0f69 in impala::Thread::SuperviseThread (name=..., category=..., 
          functor=..., thread_started=0x7f307168d8d0)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/util/thread.cc:317
      #27 0x00000000015f7f42 in boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, boost::function<void()>, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0x10f0dfc0, f=@0x10f0dfb8, a=...)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/bind/bind.hpp:457
      #28 0x00000000015f7e85 in boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::P---Type <return> to continue, or q <return> to quit---
      romise<long int>*> > >::operator()(void) (this=0x10f0dfb8)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20
      #29 0x00000000015f7de0 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x10f0de00)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0/include/boost/thread/detail/thread.hpp:116
      #30 0x0000000001a4acfa in thread_proxy ()
      #31 0x0000003864807aa1 in start_thread () from /lib64/libpthread.so.0
      #32 0x00000038640e893d in clone () from /lib64/libc.so.6
      

      From impalad.FATAL:

      F1201 03:17:57.483899 32242 mem-pool.cc:69] Check failed: chunks_.empty() Must call FreeAll() or AcquireData() for this pool
      

        Activity

        Hide
        tarasbob Taras Bobrovytsky added a comment -

        I have the relevant logs, the core file and the impalad binary.

        Show
        tarasbob Taras Bobrovytsky added a comment - I have the relevant logs, the core file and the impalad binary.
        Hide
        kwho Michael Ho added a comment -

        Is this different from IMPALA-4458 ?

        Show
        kwho Michael Ho added a comment - Is this different from IMPALA-4458 ?
        Hide
        alex.behm Alexander Behm added a comment -

        Looks like a duplicate of IMPALA-4458, but let's confirm:

        • Since there was only a single query running, is the issue easily reproducible?
        • Once you upgrade to the latest bits, this issue should be gone.
        Show
        alex.behm Alexander Behm added a comment - Looks like a duplicate of IMPALA-4458 , but let's confirm: Since there was only a single query running, is the issue easily reproducible? Once you upgrade to the latest bits, this issue should be gone.
        Hide
        dhecht Dan Hecht added a comment -

        Taras Bobrovytsky have you confirmed this is a dup?

        Show
        dhecht Dan Hecht added a comment - Taras Bobrovytsky have you confirmed this is a dup?
        Hide
        tarasbob Taras Bobrovytsky added a comment -

        Dan Hecht, not yet, I'll confirm today.

        Show
        tarasbob Taras Bobrovytsky added a comment - Dan Hecht , not yet, I'll confirm today.
        Hide
        dhecht Dan Hecht added a comment -

        Taras Bobrovytsky were you able to confirm? can we resolve this?

        Show
        dhecht Dan Hecht added a comment - Taras Bobrovytsky were you able to confirm? can we resolve this?
        Hide
        tarasbob Taras Bobrovytsky added a comment -

        I verified that this issue is fixed on a 16 node cluster. The query no longer causes a crash. I also ran the binary search part of the stress test for over 3 hours without any crashes. Impala version that was installed on the cluster:

        Server version: impalad version 2.8.0-SNAPSHOT DEBUG (build 6f730ecc3a1fb7c03cc671d5aa0a358f5766f9a9)
        
        Show
        tarasbob Taras Bobrovytsky added a comment - I verified that this issue is fixed on a 16 node cluster. The query no longer causes a crash. I also ran the binary search part of the stress test for over 3 hours without any crashes. Impala version that was installed on the cluster: Server version: impalad version 2.8.0-SNAPSHOT DEBUG (build 6f730ecc3a1fb7c03cc671d5aa0a358f5766f9a9)

          People

          • Assignee:
            tarasbob Taras Bobrovytsky
            Reporter:
            tarasbob Taras Bobrovytsky
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development