[IMPALA-6927] Crash when click Fragment Instances on web page

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.0, Impala 2.12.0
• Fix Version/s: Impala 3.0, Impala 2.13.0
• Target Version: Impala 3.0, Impala 2.13.0
• Epic Color: ghx-label-5
• Assignee: Lars Volker
• Reporter: Quanlong Huang
• Created: 25/Apr/18 13:42
• Updated: 26/Apr/18 16:40
• Resolved: 26/Apr/18 16:39

Description

When clicking on "Fragment Instances" in web page for a running query, the impalad crashed. Looks like a DCHECK was hit. In the log:

F0425 06:34:30.543802  8803 coordinator-backend-state.cc:616] Check failed: instance_stats.Size() == fragments_.size() (9 vs. 3)

The stacktrace is

#0  0x00007ff0b0ef4c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ff0b0ef8028 in __GI_abort () at abort.c:89
#2  0x0000000003fb3294 in google::DumpStackTraceAndExit() ()
#3  0x0000000003fa9d0d in google::LogMessage::Fail() ()
#4  0x0000000003fab5b2 in google::LogMessage::SendToLog() ()
#5  0x0000000003fa96e7 in google::LogMessage::Flush() ()
#6  0x0000000003faccae in google::LogMessageFatal::~LogMessageFatal() ()
#7  0x0000000002de751f in impala::Coordinator::BackendState::InstanceStatsToJson (this=0xd62ab00, value=0x7feff0f00050, document=0x7feff0f002f0)
    at /mnt/volume1/impala-orc/incubator-impala/be/src/runtime/coordinator-backend-state.cc:616
#8  0x0000000002dd7314 in impala::Coordinator::FInstanceStatsToJson (this=0xe1b4340, doc=0x7feff0f002f0) at /mnt/volume1/impala-orc/incubator-impala/be/src/runtime/coordinator.cc:973
#9  0x0000000001b20e02 in impala::ImpalaHttpHandler::QueryFInstancesHandler (this=0x73dbf70, args=..., document=0x7feff0f002f0)
    at /mnt/volume1/impala-orc/incubator-impala/be/src/service/impala-http-handler.cc:730
#10 0x0000000001b23533 in (anonymous namespace)::<lambda(const auto:1&, auto:2*)>::operator()<std::map<std::basic_string<char>, std::basic_string<char> >, rapidjson::GenericDocument<rapidjson::UTF8<> > >(const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > &, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > *) const (__closure=0xdc10238, args=..., doc=0x7feff0f002f0)
    at /mnt/volume1/impala-orc/incubator-impala/be/src/service/impala-http-handler.cc:62
#11 0x0000000001b23568 in boost::detail::function::void_function_obj_invoker2<(anonymous namespace)::MakeCallback(T*, const F&) [with T = impala::ImpalaHttpHandler; F = void (impala::ImpalaHttpHandler::*)(const std::map<std::basic_string<char>, std::basic_string<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<> >*); impala::Webserver::UrlCallback = boost::function<void(const std::map<std::basic_string<char>, std::basic_string<char> >&, rapidjson::GenericDocument<rapidjson::UTF8<> >*)>]::<lambda(const auto:1&, auto:2*)>, void, const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<const std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>::invoke(boost::detail::function::function_buffer &, const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > &, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > *) (
    function_obj_ptr=..., a0=..., a1=0x7feff0f002f0) at /mnt/volume1/impala-orc/incubator-impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#12 0x0000000001c3724a in boost::function2<void, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>::operator() (this=0xdc10230, a0=..., a1=0x7feff0f002f0)
    at /mnt/volume1/impala-orc/incubator-impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#13 0x0000000001c35599 in impala::Webserver::RenderUrlWithTemplate (this=0xebf5a40, arguments=..., url_handler=..., output=0x7feff0f008e0, content_type=0x7feff0f00a5c)
    at /mnt/volume1/impala-orc/incubator-impala/be/src/util/webserver.cc:439
#14 0x0000000001c350a6 in impala::Webserver::BeginRequestCallback (this=0xebf5a40, connection=0xd5ab000, request_info=0xd5ab000) at /mnt/volume1/impala-orc/incubator-impala/be/src/util/webserver.cc:410
#15 0x0000000001c34b47 in impala::Webserver::BeginRequestCallbackStatic (connection=0xd5ab000) at /mnt/volume1/impala-orc/incubator-impala/be/src/util/webserver.cc:369
#16 0x0000000001c47d8a in handle_request (conn=0xd5ab000) at /mnt/volume1/impala-orc/incubator-impala/be/src/thirdparty/squeasel/squeasel.c:3874
#17 0x0000000001c49fb6 in process_new_connection (conn=0xd5ab000) at /mnt/volume1/impala-orc/incubator-impala/be/src/thirdparty/squeasel/squeasel.c:4518
#18 0x0000000001c4a68a in worker_thread (thread_func_param=0xc505b80) at /mnt/volume1/impala-orc/incubator-impala/be/src/thirdparty/squeasel/squeasel.c:4650
#19 0x00007ff0b128f184 in start_thread (arg=0x7feff0f02700) at pthread_create.c:312
#20 0x00007ff0b0fbbffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

 

Not sure how to reproduce it in the minicluster. I was playing around in Impala-2.12.0-rc1 on my cluster. It can be reproduced stably. My actions are:

[localhost:21000] default> compute incremental stats myTable;

Then in the webpage, the following subquery will be running:

SELECT COUNT(*), realdate FROM myTable GROUP BY realdate

Click its details, then click the "Fragment Instances" tab and the impalad crashed.