[IMPALA-5457] OOM during profile serialization leads to crash

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: crash
• Epic Color: ghx-label-3
• Assignee: Unassigned
• Reporter: Lars Volker
• Created: 08/Jun/17 03:38
• Updated: 27/Jun/17 18:23
• Resolved: 27/Jun/17 18:23

Description

It looks like an oom exception during thrift serialization is not handled correctly and can lead to a crash. Here is the stack trace of the crashing thread from a minidump.

 0  libc-2.12.so!__GI_raise + 0x35
 1  libc-2.12.so!__GI_abort + 0x175
 2  libstdc++.so.6!__gnu_cxx::__verbose_terminate_handler() + 0x15d
 3  libstdc++.so.6!std::rethrow_exception(std::__exception_ptr::exception_ptr) + 0x76
 4  libstdc++.so.6!std::terminate() + 0x11
 5  libstdc++.so.6!__cxa_throw + 0x68
 6  impalad!(anonymous namespace)::handle_oom(void* (*)(void*), void*, bool, bool) [clone .constprop.53] + 0x3a8
 7  impalad!operator new(unsigned long) + 0x2b0
 8  impalad!std::vector<impala::TRuntimeProfileNode, std::allocator<impala::TRuntimeProfileNode> >::reserve(unsigned long) [new_allocator.h : 104 + 0x10]
 9  impalad!impala::RuntimeProfile::ToThrift(std::vector<impala::TRuntimeProfileNode, std::allocator<impala::TRuntimeProfileNode> >*) const [runtime-profile.cc : 755 + 0x8]
10  impalad!impala::RuntimeProfile::SerializeToArchiveString(std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*) const [runtime-profile.cc : 724 + 0x5]
11  impalad!impala::ImpalaServer::GetRuntimeProfileStr(impala::TUniqueId const&, bool, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*) [impala-server.cc : 571 + 0x5]
12  impalad!impala::ImpalaHttpHandler::QueryProfileEncodedHandler(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*) [impala-http-handler.cc : 219 + 0x1c]
13  impalad!impala::Webserver::RenderUrlWithTemplate(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, impala::Webserver::UrlHandler const&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, impala::ContentType*) [function_template.hpp : 767 + 0x11]
14  impalad!impala::Webserver::BeginRequestCallback(sq_connection*, sq_request_info*) [webserver.cc : 418 + 0x1c]
15  impalad!handle_request + 0x740
16  impalad!process_new_connection + 0x16d
17  impalad!worker_thread + 0x2dd
18  libpthread-2.12.so!start_thread + 0xd1
19  libc-2.12.so!clone + 0x6d

Issue Links

• duplicates: IMPALA-5487 Race in runtime-profile.cc::toThrift() can lead to corrupt profiles being generated while query is running