[IMPALA-5102] Handle uncaught exceptions in Impalad

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Later
• Affects Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: crash
• Target Version: Impala 2.13.0
• Epic Color: ghx-label-2
• Assignee: Unassigned
• Reporter: Michael Ho
• Created: 20/Mar/17 21:37
• Updated: 08/Oct/20 23:52
• Resolved: 08/Oct/20 23:52

Description

Impalad uses noexcept API of boost library whenever possible. However, there are certain API which don't implement the noexcept variant. One example of this is the thread creation interface:

void Thread::StartThread(const ThreadFunctor& functor) {
  DCHECK(thread_manager.get() != nullptr)
      << "Thread created before InitThreading called";
  DCHECK(tid_ == UNINITIALISED_THREAD_ID) << "StartThread called twice";

  Promise<int64_t> thread_started;
  thread_.reset(
      new thread(&Thread::SuperviseThread, name_, category_, functor, &thread_started));

  // TODO: This slows down thread creation although not enormously. To make this faster,
  // consider delaying thread_started.Get() until the first call to tid(), but bear in
  // mind that some coordination is required between SuperviseThread() and this to make
  // sure that the thread is still available to have its tid set.
  tid_ = thread_started.Get();

  VLOG(2) << "Started thread " << tid_ << " - " << category_ << ":" << name_;
}

We have been bitten by this uncaught exception in the past such as IMPALA-3104. This kind of exception is more prone to occur when there are a large number of fragment instances running in an Impala cluster.

Thread creation exception handling is dealt with in IMPALA-5750.

There are other uncaught exceptions in the code. Please update this JIRA as we find more.

Issue Links

• is cloned by: IMPALA-5750 Handle uncaught exceptions in thread creation
• is related to: IMPALA-3290 Catch std::bad_alloc from LLVM optimisation