[IMPALA-2606] exec-node.cc:190] Check failed: mem_tracker()->consumption() == 0 (16777216 vs. 0) Leaked memory.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.5.0
• Component/s: Backend
• Labels: resource-management stress
• Environment: impalad version 2.3.0-cdh5.5.0 DEBUG (build dff9ea16a211e1dd069039d2c7e769391223ca2f); Built on Wed, 21 Oct 2015 22:01:51 PST
• Target Version: Impala 2.5.0
• Assignee: Daniel Hecht
• Reporter: casey
• Created: 29/Oct/15 05:09
• Updated: 03/Jun/16 03:40
• Resolved: 04/Feb/16 00:28

Description

The dheck below failed in a stress test run. The build it failed on is a week old and its been running most of the time so this seems rare.

Fatal log:

F1028 20:25:53.754781 12298 exec-node.cc:190] Check failed: mem_tracker()->consumption() == 0 (16777216 vs. 0) Leaked memory.
Fragment c143ca7328441104:a63c67f8783e3db9: Consumption=16.02 MB
  AGGREGATION_NODE (id=1): Consumption=16.00 MB
  HDFS_SCAN_NODE (id=0): Consumption=0
  DataStreamSender: Consumption=20.56 KB

Backtrace:

#6  0x000000000205dd2d in google::LogMessageFatal::~LogMessageFatal (this=0x7f7e309dbe90, __in_chrg=<value optimized out>) at src/logging.cc:1836
#7  0x0000000001597078 in impala::ExecNode::Close (this=0x1d9a02900, state=0x16222d00) at /usr/src/debug/impala-2.3.0-cdh5.5.0/be/src/exec/exec-node.cc:191
#8  0x000000000166e46f in impala::PartitionedAggregationNode::Close (this=0x1d9a02900, state=0x16222d00) at /usr/src/debug/impala-2.3.0-cdh5.5.0/be/src/exec/partitioned-aggregation-node.cc:433
#9  0x00000000015602c1 in impala::PlanFragmentExecutor::Close (this=0xa8b52628) at /usr/src/debug/impala-2.3.0-cdh5.5.0/be/src/runtime/plan-fragment-executor.cc:573
#10 0x0000000001331392 in impala::FragmentMgr::FragmentExecState::Exec (this=0xa8b52400) at /usr/src/debug/impala-2.3.0-cdh5.5.0/be/src/service/fragment-exec-state.cc:51
#11 0x0000000001329994 in impala::FragmentMgr::FragmentExecThread (this=0x7d1b140, exec_state=0xa8b52400) at /usr/src/debug/impala-2.3.0-cdh5.5.0/be/src/service/fragment-mgr.cc:83
#12 0x000000000132d384 in boost::_mfi::mf1<void, impala::FragmentMgr, impala::FragmentMgr::FragmentExecState*>::operator() (this=0x24fff33e0, p=0x7d1b140, a1=0xa8b52400)
    at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/mem_fn_template.hpp:165
#13 0x000000000132d13d in boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::FragmentMgr::FragmentExecState*> >::operator()<boost::_mfi::mf1<void, impala::FragmentMgr, impala::FragmentMgr::FragmentExecState*>, boost::_bi::list0> (this=0x24fff33f0, f=..., a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:313
#14 0x000000000132ca25 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::FragmentMgr::FragmentExecState*>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::FragmentMgr::FragmentExecState*> > >::operator() (this=0x24fff33e0) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
#15 0x000000000132c373 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::FragmentMgr::FragmentExecState*>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::FragmentMgr::FragmentExecState*> > >, void>::invoke (function_obj_ptr=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:153
#16 0x00000000011f800d in boost::function0<void>::operator() (this=0x7f7e309dccc0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:767
#17 0x0000000001412f0a in impala::Thread::SuperviseThread (name="exec-plan-fragment", category="impala-server", functor=..., thread_started=0x7f7d62a70a40)
    at /usr/src/debug/impala-2.3.0-cdh5.5.0/be/src/util/thread.cc:314

Issue Links

• is duplicated by: IMPALA-3665 [exec-node.cc:173] Query 6435672900aad9b:701ec2b9768976bb leaked memory.
• relates to: IMPALA-1867 Crash: DCHECK: mem_tracker()->consumption() == 0 (8388608 vs. 0) Leaked memory.