[IMPALA-10537] Uncaught exception in ThriftDebugString can crash catalog server

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Epic Color: ghx-label-2
• Assignee: Vihang Karajgaonkar
• Reporter: Vihang Karajgaonkar
• Created: 22/Feb/21 23:28
• Updated: 22/Feb/21 23:43
• Resolved: 22/Feb/21 23:43

Description

The catalogd debug UI provides a way to display a textual representation of a catalog object. However, if the object is too large or if there are any exceptions during the deserialization the thrift library can throw a TProtocolException or bad_alloc exception. These exceptions when remain uncaught and crash the catalog server with the following trace in the minidump.

Operating system: Linux
                  0.0.0 Linux 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64
CPU: amd64
     family 6 model 79 stepping 1
     1 CPU

GPU: UNKNOWN

Crash reason:  SIGABRT
Crash address: 0x3d40000db31
Process uptime: not available

Thread 161 (crashed)
 0  libc-2.17.so + 0x36207
 1  libc-2.17.so + 0x378f8
 2  libstdc++.so.6!std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::basic_string<wchar_t const*>(wchar_t const*, wchar_t const*, std::allocator<wchar_t> const&) + 0x142
 3  libstdc++.so.6!std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::basic_string<wchar_t const*>(wchar_t const*, wchar_t const*, std::allocator<wchar_t> const&) + 0x142
 4  libstdc++.so.6!__gnu_cxx::__verbose_terminate_handler() + 0x15d
 5  libstdc++.so.6!std::rethrow_exception(std::__exception_ptr::exception_ptr) + 0x76
 6  libstdc++.so.6!std::terminate() + 0x11
 7  libstdc++.so.6!__cxa_throw + 0x69
 8  impalad!apache::thrift::transport::TMemoryBuffer::ensureCanWrite(unsigned int) + 0x179
 9  impalad!apache::thrift::transport::TMemoryBuffer::writeSlow(unsigned char const*, unsigned int) + 0x13
10  impalad!apache::thrift::protocol::TDebugProtocol::writePlain(std::string const&) + 0x1f
11  impalad!apache::thrift::protocol::TDebugProtocol::writeItem(std::string const&) + 0x1c
12  impalad!apache::thrift::protocol::TDebugProtocol::writeString(std::string const&) + 0x3c8
13  impalad!impala::THdfsFileDesc::write(apache::thrift::protocol::TProtocol*) const + 0x62
14  impalad!impala::THdfsPartition::write(apache::thrift::protocol::TProtocol*) const + 0x3a3
15  impalad!impala::THdfsTable::write(apache::thrift::protocol::TProtocol*) const + 0x1a1
16  impalad!impala::TTable::write(apache::thrift::protocol::TProtocol*) const + 0x421
17  impalad!impala::TCatalogObject::write(apache::thrift::protocol::TProtocol*) const + 0x2bc
18  impalad!std::string apache::thrift::ThriftDebugString<impala::TCatalogObject>(impala::TCatalogObject const&) + 0x9b
19  impalad!impala::CatalogServer::CatalogObjectsUrlCallback(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>*) + 0xea8
20  impalad!impala::Webserver::RenderUrlWithTemplate(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, impala::Webserver::UrlHandler const&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, impala::ContentType*) + 0x170
21  impalad!impala::Webserver::BeginRequestCallback(sq_connection*, sq_request_info*) + 0x2b4
22  impalad!handle_request + 0x740
23  impalad!process_new_connection + 0x16d
24  impalad!worker_thread + 0x2e5
25  libpthread-2.17.so + 0x7dd5
26  libc-2.17.so + 0xfdead

Issue Links

• duplicates: IMPALA-10450 Catalogd crashes in serializing thrift debug string