[IMPALA-3589] impala query exception cause some instance not free memery then memery used over 100G , then always mem_limit error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 2.6.0
• Component/s: Backend
• Environment: use gperftools-2.5
• Assignee: Tim Armstrong
• Reporter: fishing
• Created: 22/May/16 02:57
• Updated: 31/May/16 01:15
• Resolved: 27/May/16 17:17

Description

Thread 803 (Thread 0x7f1beea42700 (LWP 5860)):
#0 PackedCache<35, unsigned long>::KeyMatch (key=96132, entry=12304956) at src/packed-cache-inl.h:217
#1 PackedCache<35, unsigned long>::GetOrDefault (default_value=0, key=96132, this=0x2fe4010) at src/packed-cache-inl.h:175
#2 tcmalloc::PageHeap::GetSizeClassIfCached (p=96132, this=0x2fe4000) at src/page_heap.h:191
#3 (anonymous namespace)::do_free_helper (invalid_free_fn=0x17dccd0 <(anonymous namespace)::InvalidFree(void*)>, size_hint=0, use_hint=false, heap_must_be_valid=true, heap=0x767c9680, ptr=0x2ef08000) at src/tcmalloc.cc:1282
#4 (anonymous namespace)::do_free_with_callback (invalid_free_fn=0x17dccd0 <(anonymous namespace)::InvalidFree(void*)>, size_hint=0, use_hint=false, ptr=0x2ef08000) at src/tcmalloc.cc:1337
#5 (anonymous namespace)::do_free (ptr=0x2ef08000) at src/tcmalloc.cc:1345
#6 tc_free (ptr=0x2ef08000) at src/tcmalloc.cc:1610
#7 0x0000000000a8f80a in impala::MemPool::FreeAll (this=this@entry=0x7f1beea40fd8) at /export/ldb/Impala-cdh5.7/be/src/runtime/mem-pool.cc:86
#8 0x0000000000acd54d in impala::RowBatch::Reset (this=this@entry=0x7f1beea40fb0) at /export/ldb/Impala-cdh5.7/be/src/runtime/row-batch.cc:341
#9 0x0000000000ccd898 in impala::PartitionedAggregationNode::Open (this=0x1cde16c0, state=0x7f1a70cf0d00) at /export/ldb/Impala-cdh5.7/be/src/exec/partitioned-aggregation-node.cc:342
#10 0x0000000000dfb5cc in impala::PlanFragmentExecutor::OpenInternal (this=this@entry=0x7f18fabca690) at /export/ldb/Impala-cdh5.7/be/src/runtime/plan-fragment-executor.cc:374
#11 0x0000000000dfbe23 in impala::PlanFragmentExecutor::Open (this=this@entry=0x7f18fabca690) at /export/ldb/Impala-cdh5.7/be/src/runtime/plan-fragment-executor.cc:341
#12 0x0000000000b84a48 in impala::FragmentMgr::FragmentExecState::Exec (this=0x7f18fabca400) at /export/ldb/Impala-cdh5.7/be/src/service/fragment-exec-state.cc:56
#13 0x0000000000b7d34a in impala::FragmentMgr::FragmentThread (this=0x93a85e0, fragment_instance_id=...) at /export/ldb/Impala-cdh5.7/be/src/service/fragment-mgr.cc:86
#14 0x0000000000b7f5d0 in std::Mem_fn<void (impala::FragmentMgr::*)(impala::TUniqueId)>::operator()<impala::TUniqueId, void> (_object=<optimized out>, this=0xad40ca0) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:569
#15 std::Bind<std::_Mem_fn<void (impala::FragmentMgr::)(impala::TUniqueId)>(impala::FragmentMgr, std::_Placeholder<1>)>::call<void, impala::TUniqueId&&, 0ul, 1ul> (_args=..., this=0xad40ca0) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:1264
#16 std::_Bind<std::_Mem_fn<void (impala::FragmentMgr::)(impala::TUniqueId)>(impala::FragmentMgr, std::_Placeholder<1>)>::operator()<impala::TUniqueId, void> (this=0xad40ca0) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:1323
#17 std::Function_handler<void (impala::TUniqueId), std::_Bind<std::_Mem_fn<void (impala::FragmentMgr::)(impala::TUniqueId)> (impala::FragmentMgr, std::_Placeholder<1>)> >::_M_invoke(std::_Any_data const&, impala::TUniqueId) (_functor=..., __args#0=...) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:2039
#18 0x0000000000b7d3f4 in std::function<void (impala::TUniqueId)>::operator()(impala::TUniqueId) const (__args#0=..., this=0x93a8618) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:2439
#19 impala::FragmentMgr::<lambda()>::operator() (__closure=<optimized out>) at /export/ldb/Impala-cdh5.7/be/src/service/fragment-mgr.cc:78
#20 std::Function_handler<void(), impala::FragmentMgr::ExecPlanFragment(const impala::TExecPlanFragmentParams&)::<lambda()> >::_M_invoke(const std::_Any_data &) (_functor=...) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:2039
#21 0x0000000000b7e9c2 in std::function<void ()>::operator()() const (this=0x7f18facf4b40) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:2439
#22 std::future<void> threadpool11::Pool::postWork<void>(std::function<void ()>, threadpool11::Work::Type)::

{lambda()#1}::operator()() (__closure=0x7f18facf4b40) at /export/ldb/Impala-cdh5.7/be/src/threadpool11/pool.hpp:259
#23 std::_Function_handler<threadpool11::Work::Type (), std::future<void> threadpool11::Pool::postWork<void>(std::function<void ()>, threadpool11::Work::Type)::{lambda()#1}

>::_M_invoke(std::Any_data const&) (_functor=...) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:2025
#24 0x0000000000c38768 in std::function<threadpool11::Work::Type ()>::operator()() const (this=0xa5d0040) at /usr/local/gcc-4.9.3/include/c++/4.9.3/functional:2439
#25 threadpool11::Worker::execute (this=0x37916c8, pool=...) at /export/ldb/Impala-cdh5.7/be/src/threadpool11/worker.cpp:49
#26 0x00007f1d0b3bb680 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../gcc-4.9.3/libstdc+-v3/src/c+11/thread.cc:84
#27 0x00000035d60079d1 in start_thread () from /export/servers/impala/lib_C/libpthread.so.0
#28 0x00000032ca0e8b6d in clone () from /export/servers/impala/lib_C/libc.so.6

Issue Links

• duplicates: IMPALA-3582 Impala appears to be leaking memory in stress test