[IMPALA-6274] Impala daemon crashing in Coordinator::Wait() or some of its callees

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Labels: crash
• Environment: RHEL 7.2
• Target Version: Impala 3.0, Impala 2.12.0
• Epic Color: ghx-label-6
• Assignee: Unassigned
• Reporter: Zoram Thanga
• Created: 05/Dec/17 17:57
• Updated: 06/Apr/18 23:42
• Resolved: 06/Apr/18 23:42

Description

Impala daemon is crashing quite frequently with the following stack traces, which points to

(a) Accessing a stale QueryExecState::coord_, or
(b) Some sort of memory corruption, or
(c) Both (a) and (b)

EDIT: Delete irrelevant noise to aid readability.

Stack trace 1
Thread 632 (crashed)
0 libc-2.17.so + 0x351d7
1 libc-2.17.so + 0x368c8
2 ld-2.17.so + 0x150c0
3 libstdc++.so.6!<name omitted> [char_traits.h : 275 + 0x8]
4 0x7f02acec01c0
5 libstdc++.so.6!<name omitted> [vterminate.cc : 95 + 0x5]
6 libstdc++.so.6!<name omitted> [eh_terminate.cc : 47 + 0x2]
7 libstdc++.so.6!<name omitted> [eh_terminate.cc : 57 + 0xd]
8 libstdc++.so.6!<name omitted> [eh_throw.cc : 87 + 0x5]
10 impalad!<name omitted> [mutex.hpp : 119 + 0x5]
11 impalad!impala::Coordinator::UpdateStatus(impala::Status const&, impala::TUniqueId const&, std::string const&) [lock_guard.hpp : 38 + 0x8]
12 impalad!impala::Coordinator::Wait() [coordinator.cc : 1076 + 0x14]
13 impalad!impala::ImpalaServer::QueryExecState::WaitInternal() [query-exec-state.cc : 639 + 0x8]
14 impalad!impala::ImpalaServer::QueryExecState::Wait() [query-exec-state.cc : 607 + 0xb]
15 impalad!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) [function_template.hpp : 767 + 0x7]
16 impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void (std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>> > > >::run() [bind.hpp : 457 + 0x6]
17 impalad!thread_proxy + 0xda

Issue Links

• Is contained by: IMPALA-2550 Switch to per-query exec rpc