Details
-
Sub-task
-
Status: Resolved
-
Critical
-
Resolution: Not A Bug
-
None
-
None
-
None
-
None
-
ghx-label-9
Description
Queries with a larger number of fragments can fail with
Timed out: ReportExecStatus RPC to 10.17.187.36:22000 timed out after 10.000s (SENT)
Vtune shows that while deserializing the query profile the thread can get stuck in tcmalloc
impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file] impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source file] impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file] impalad ! operator new + 0x297 - [unknown source file] impalad ! __gnu_cxx::new_allocator<impala::TCounter>::allocate + 0x4 - new_allocator.h:104 impalad ! std::vector<impala::TCounter, std::allocator<impala::TCounter>>::resize + 0x7f3 - stl_vector.h:676 impalad ! impala::TRuntimeProfileNode::read + 0xe5a - RuntimeProfile_types.cpp:601 impalad ! impala::TRuntimeProfileTree::read + 0x8ac - RuntimeProfile_types.cpp:982 impalad ! impala::TReportExecStatusParams::read + 0x156 - ImpalaInternalService_types.cpp:2956 impalad ! impala::DeserializeThriftMsg<impala::TReportExecStatusParams> + 0xe5 - thrift-util.h:145 impalad ! impala::DeserializeFromSidecar<impala::TReportExecStatusParams> + 0xb7 - rpc.h:407 impalad ! impala::ExecControlService::ReportExecStatus + 0x21e - impala-internal-service.cc:148 impalad ! std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator() + 0x1c - functional:2439 impalad ! kudu::rpc::GeneratedServiceIf::Handle + 0x188 - service_if.cc:134 impalad ! impala::ImpalaServicePool::RunThread + 0x241 - impala-service-pool.cc:130 impalad ! boost::function0<void>::operator() + 0x1a - function_template.hpp:767 impalad ! impala::Thread::SuperviseThread + 0x20e - thread.cc:325 impalad ! operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0> + 0x5a - bind.hpp:457 impalad ! boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>::operator() - bind_template.hpp:20 impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x19 - thread.hpp:116 impalad ! thread_proxy + 0xd9 - [unknown source file] libpthread.so.0 ! start_thread + 0xd0 - [unknown source file] libc.so.6 ! clone + 0x6c - [unknown source file]
