[IMPALA-2123] BufferedBlockMgr crash in fragment close while writing

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.2
• Labels: crash resource-management
• Target Version: Impala 2.3.0
• Assignee: Ippokratis Pandis
• Reporter: Matthew Jacobs
• Created: 06/Jul/15 16:26
• Updated: 20/Dec/15 00:05
• Resolved: 09/Sep/15 03:28

Description

A exhaustive run on the 2.2.0 build failed because an Impalad crashed while tearing down a fragment. It looks like the fragment had an outstanding write, but clearly some state wasn't cleaned up correctly because WriteComplete() crashed. Unfortunately I don't have a better way to reproduce this.

Stack: [0x00007f4cacdbe000,0x00007f4cad7bf000],  sp=0x00007f4cad7bc390,  free space=10232k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [impalad+0xd7416c]  std::less<impala::TErrorCode::type>::operator()(impala::TErrorCode::type const&, impala::TErrorCode::type const&) const+0x14
C  [impalad+0xd74e58]  std::_Rb_tree<impala::TErrorCode::type, std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry>, std::_Select1st<std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry> >, std::less<impala::TErrorCode::type>, std::allocator<std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry> > >::_M_lower_bound(std::_Rb_tree_node<std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry> > const*, std::_Rb_tree_node<std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry> > const*, impala::TErrorCode::type const&) const+0x3c
C  [impalad+0xd742ae]  std::_Rb_tree<impala::TErrorCode::type, std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry>, std::_Select1st<std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry> >, std::less<impala::TErrorCode::type>, std::allocator<std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry> > >::find(impala::TErrorCode::type const&) const+0x4a
C  [impalad+0xd73f73]  std::map<impala::TErrorCode::type, impala::TErrorLogEntry, std::less<impala::TErrorCode::type>, std::allocator<std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry> > >::find(impala::TErrorCode::type const&) const+0x23
C  [impalad+0xd73977]  impala::ErrorCount(std::map<impala::TErrorCode::type, impala::TErrorLogEntry, std::less<impala::TErrorCode::type>, std::allocator<std::pair<impala::TErrorCode::type const, impala::TErrorLogEntry> > > const&)+0x26
C  [impalad+0xb7d5cd]  impala::RuntimeState::LogError(impala::ErrorMsg const&)+0x177
C  [impalad+0xb9cf9a]  impala::BufferedBlockMgr::WriteComplete(impala::BufferedBlockMgr::Block*, impala::Status const&)+0xb72
C  [impalad+0xbac418]  boost::_mfi::mf2<void, impala::BufferedBlockMgr, impala::BufferedBlockMgr::Block*, impala::Status const&>::operator()(impala::BufferedBlockMgr*, impala::BufferedBlockMgr::Block*, impala::Status const&) const+0x70
C  [impalad+0xbac0ad]  void boost::_bi::list3<boost::_bi::value<impala::BufferedBlockMgr*>, boost::_bi::value<impala::BufferedBlockMgr::Block*>, boost::arg<1> >::operator()<boost::_mfi::mf2<void, impala::BufferedBlockMgr, impala::BufferedBlockMgr::Block*, impala::Status const&>, boost::_bi::list1<impala::Status const&> >(boost::_bi::type<void>, boost::_mfi::mf2<void, impala::BufferedBlockMgr, impala::BufferedBlockMgr::Block*, impala::Status const&>&, boost::_bi::list1<impala::Status const&>&, int)+0x87
C  [impalad+0xbab8b6]  void boost::_bi::bind_t<boost::_bi::unspecified, boost::_mfi::mf2<void, impala::BufferedBlockMgr, impala::BufferedBlockMgr::Block*, impala::Status const&>, boost::_bi::list3<boost::_bi::value<impala::BufferedBlockMgr*>, boost::_bi::value<impala::BufferedBlockMgr::Block*>, boost::arg<1> > >::operator()<impala::Status>(impala::Status const&)+0x4a
C  [impalad+0xbaaf2c]  boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<boost::_bi::unspecified, boost::_mfi::mf2<void, impala::BufferedBlockMgr, impala::BufferedBlockMgr::Block*, impala::Status const&>, boost::_bi::list3<boost::_bi::value<impala::BufferedBlockMgr*>, boost::_bi::value<impala::BufferedBlockMgr::Block*>, boost::arg<1> > >, void, impala::Status const&>::invoke(boost::detail::function::function_buffer&, impala::Status const&)+0x2e
C  [impalad+0xbbfc22]  boost::function1<void, impala::Status const&>::operator()(impala::Status const&) const+0x7e
C  [impalad+0xbc5cf8]  impala::DiskIoMgr::RequestContext::Cancel(impala::Status const&)+0x7ac
C  [impalad+0xbb6388]  impala::DiskIoMgr::CancelContext(impala::DiskIoMgr::RequestContext*, bool)+0x30
C  [impalad+0xbb5fb0]  impala::DiskIoMgr::UnregisterContext(impala::DiskIoMgr::RequestContext*)+0x3c
C  [impalad+0xb99477]  impala::BufferedBlockMgr::~BufferedBlockMgr()+0x1f9
C  [impalad+0xbaa53a]  void boost::checked_delete<impala::BufferedBlockMgr>(impala::BufferedBlockMgr*)+0x1e
C  [impalad+0xbac836]  boost::detail::sp_counted_impl_p<impala::BufferedBlockMgr>::dispose()+0x1c
C  [impalad+0xaa7a46]  boost::detail::sp_counted_base::release()+0x42
C  [impalad+0xaa7abf]  boost::detail::shared_count::~shared_count()+0x27
C  [impalad+0xb8071c]  boost::shared_ptr<impala::BufferedBlockMgr>::~shared_ptr()+0x1c
C  [impalad+0xb8133f]  boost::shared_ptr<impala::BufferedBlockMgr>::reset()+0x5b
C  [impalad+0xb7bea7]  impala::RuntimeState::~RuntimeState()+0x21
C  [impalad+0xcc4de7]  void boost::checked_delete<impala::RuntimeState>(impala::RuntimeState*)+0x1e
C  [impalad+0xcc1305]  boost::scoped_ptr<impala::RuntimeState>::~scoped_ptr()+0x1b
C  [impalad+0x10a5165]  impala::PlanFragmentExecutor::~PlanFragmentExecutor()+0x18d
C  [impalad+0xcdd999]  impala::FragmentMgr::FragmentExecState::~FragmentExecState()+0xe9
C  [impalad+0xcdeb82]  void boost::checked_delete<impala::FragmentMgr::FragmentExecState>(impala::FragmentMgr::FragmentExecState*)+0x1e
C  [impalad+0xce109a]  boost::detail::sp_counted_impl_p<impala::FragmentMgr::FragmentExecState>::dispose()+0x1c
C  [impalad+0xaa7a46]  boost::detail::sp_counted_base::release()+0x42
C  [impalad+0xaa7abf]  boost::detail::shared_count::~shared_count()+0x27
C  [impalad+0xc1ca0a]  boost::shared_ptr<impala::FragmentMgr::FragmentExecState>::~shared_ptr()+0x1c
C  [impalad+0xcdc611]  impala::FragmentMgr::FragmentExecThread(impala::FragmentMgr::FragmentExecState*)+0x359

Jenkins failure:
http://sandbox.jenkins.cloudera.com/job/impala-cdh5-2.2.0_5.4.x-exhaustive/121/

Issue Links

• is duplicated by: IMPALA-2252 Crash (likely race) tearing down BufferedBlockMgr on query failure