Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-768

Possible self-deadlock in RaftConsensus::Shutdown()

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • M5
    • None
    • consensus
    • None

    Description

      1. ASAN build
      2. RUN:
      client-test --gtest_filter=ClientTest.TestScanFaultTolerance --gtest_repeat=10000 --gtest_break_on_failure
      3: Also RUN:
      stress -c 8 -i 2 -m 2

      Result:

      I0514 17:21:13.264490 13059 tablet_server.cc:126] TabletServer shut down complete. Bye!
I0514 17:21:13.264513 13059 tablet_server.cc:117] TabletServer shutting down...
I0514 17:21:13.264566 13059 tablet_server.cc:126] TabletServer shut down complete. Bye!
I0514 17:21:13.267006 13969 raft_consensus.cc:1409] T e29170f064564f5dacf14f0f0da9040b P b924c2be3758405ba01116f291d19b92 [term 3 FOLLOWER]: Snoozing failure detection for an additional: 1.500s
I0514 17:21:13.267066 13969 raft_consensus.cc:1298] T e29170f064564f5dacf14f0f0da9040b P b924c2be3758405ba01116f291d19b92 [term 3 FOLLOWER]: Received election callback for term 3 while not running: Illegal state: Unable to lock ReplicaState for config change: State = 3
I0514 17:21:13.342953 13059 tablet_server.cc:117] TabletServer shutting down...
F0514 17:21:13.267776 13969 mutex.cc:59] Check failed: 0 == rv (0 vs. 35) . Resource deadlock avoided. Owner tid: 881; Self tid: 881; Owner stack:
    @     0x7f22901db7ad  kudu::Mutex::CheckUnheldAndMark() at ??:0
    @     0x7f22901dba03  kudu::Mutex::Acquire() at ??:0
    @     0x7f2290249eb8  kudu::MutexLock::Lock() at ??:0
    @     0x7f2290248062  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f229024b02c  boost::_bi::bind_t<>::operator()() at ??:0
    @     0x7f2291459fbb  boost::function0<>::operator()() at ??:0
    @     0x7f22902330c8  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f2292b89182  start_thread at ??:0
    @     0x7f228e90f47d  (unknown) at ??:0
    @              (nil)  (unknown)
*** Check failure stack trace: ***
(gdb) bt
 #0  0x00007fc9276d0cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
 #1  0x00007fc9276d40d8 in __GI_abort () at abort.c:89
 #2  0x0000000000d98c39 in google::DumpStackTraceAndExit () at src/utilities.cc:147
 #3  0x0000000000d8fead in google::LogMessage::Fail () at src/logging.cc:1458
 #4  0x0000000000d91d4d in google::LogMessage::SendToLog (this=0x7fc907f00450) at src/logging.cc:1412
 #5  0x0000000000d8fa9c in google::LogMessage::Flush (this=this@entry=0x7fc907f00450) at src/logging.cc:1281
 #6  0x0000000000d9266e in google::LogMessageFatal::~LogMessageFatal (this=0x7fc907f00450, __in_chrg=<optimized out>) at src/logging.cc:1984
 #7  0x0000000001ac3d45 in kudu::Mutex::Acquire (this=0x36152e8) at src/kudu/util/mutex.cc:59
 #8  0x0000000000d041ae in kudu::MutexLock::MutexLock (this=0x7fc907f004d0, lock=...) at src/kudu/util/mutex.h:76
 #9  0x0000000001ae9aae in kudu::ThreadPool::Shutdown (this=0x36152c0) at src/kudu/util/threadpool.cc:132
 #10 0x0000000001ae95f5 in kudu::ThreadPool::~ThreadPool (this=0x36152c0, __in_chrg=<optimized out>) at src/kudu/util/threadpool.cc:102
 #11 0x0000000000dff4d2 in kudu::DefaultDeleter<kudu::ThreadPool>::operator() (this=0x4035208, ptr=0x36152c0) at src/kudu/gutil/gscoped_ptr.h:145
 #12 0x0000000000df761c in kudu::internal::gscoped_ptr_impl<kudu::ThreadPool, kudu::DefaultDeleter<kudu::ThreadPool> >::~gscoped_ptr_impl (this=0x4035208, __in_chrg=<optimized out>) at src/kudu/gutil/gscoped_ptr.h:228
 #13 0x0000000000df1844 in gscoped_ptr<kudu::ThreadPool, kudu::DefaultDeleter<kudu::ThreadPool> >::~gscoped_ptr (this=0x4035208, __in_chrg=<optimized out>) at src/kudu/gutil/gscoped_ptr.h:318
 #14 0x0000000001003c9a in kudu::consensus::RaftConsensus::~RaftConsensus (this=0x40351e0, __in_chrg=<optimized out>) at src/kudu/consensus/raft_consensus.cc:154
 #15 0x0000000001003dfc in kudu::consensus::RaftConsensus::~RaftConsensus (this=0x40351e0, __in_chrg=<optimized out>) at src/kudu/consensus/raft_consensus.cc:156
 #16 0x0000000000f0d117 in kudu::RefCountedThreadSafe<kudu::consensus::Consensus, kudu::DefaultRefCountedThreadSafeTraits<kudu::consensus::Consensus> >::DeleteInternal (x=0x40351e0) at src/kudu/gutil/ref_counted.h:152
 #17 0x0000000000f0c872 in kudu::DefaultRefCountedThreadSafeTraits<kudu::consensus::Consensus>::Destruct (x=0x40351e0) at src/kudu/gutil/ref_counted.h:115
 #18 0x0000000000f0ba80 in kudu::RefCountedThreadSafe<kudu::consensus::Consensus, kudu::DefaultRefCountedThreadSafeTraits<kudu::consensus::Consensus> >::Release (this=0x40351e8) at src/kudu/gutil/ref_counted.h:143
 #19 0x00000000010179b2 in kudu::internal::MaybeRefcount<true, kudu::consensus::RaftConsensus*>::Release (o=0x40351e0) at src/kudu/gutil/bind_helpers.h:469
 #20 0x00000000010177fa in kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::consensus::RaftConsensus::*)(kudu::consensus::ElectionResult const&)>, void (kudu::consensus::RaftConsensus*, kudu::consensus::ElectionResult const&), void (kudu::consensus::RaftConsensus*, kudu::consensus::ElectionResult)>::~BindState() (this=0x38f7220, __in_chrg=<optimized out>) at src/kudu/gutil/bind_internal.h:2493
 #21 0x000000000101787c in kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::consensus::RaftConsensus::*)(kudu::consensus::ElectionResult const&)>, void (kudu::consensus::RaftConsensus*, kudu::consensus::ElectionResult const&), void (kudu::consensus::RaftConsensus*, kudu::consensus::ElectionResult)>::~BindState() (this=0x38f7220, __in_chrg=<optimized out>) at src/kudu/gutil/bind_internal.h:2494
 #22 0x0000000000d4b41e in kudu::RefCountedThreadSafe<kudu::internal::BindStateBase, kudu::DefaultRefCountedThreadSafeTraits<kudu::internal::BindStateBase> >::DeleteInternal (x=0x38f7220) at src/kudu/gutil/ref_counted.h:152
 #23 0x0000000000d495cf in kudu::DefaultRefCountedThreadSafeTraits<kudu::internal::BindStateBase>::Destruct (x=0x38f7220) at src/kudu/gutil/ref_counted.h:115
 #24 0x0000000000d46ede in kudu::RefCountedThreadSafe<kudu::internal::BindStateBase, kudu::DefaultRefCountedThreadSafeTraits<kudu::internal::BindStateBase> >::Release (this=0x38f7228) at src/kudu/gutil/ref_counted.h:143
 #25 0x0000000001afa46f in scoped_refptr<kudu::internal::BindStateBase>::~scoped_refptr (this=0x2f5d450, __in_chrg=<optimized out>) at src/kudu/gutil/ref_counted.h:251
 #26 0x0000000001afa406 in kudu::internal::CallbackBase::~CallbackBase (this=0x2f5d450, __in_chrg=<optimized out>) at src/kudu/gutil/callback_internal.cc:32
 #27 0x0000000000df1878 in kudu::Callback<void ()>::~Callback() (this=0x2f5d450, __in_chrg=<optimized out>) at src/kudu/gutil/callback.h:366
 #28 0x0000000000ff5dba in boost::_bi::value<kudu::Callback<void ()> >::~value() (this=0x2f5d450, __in_chrg=<optimized out>) at /usr/include/boost/bind/bind.hpp:112
 #29 0x0000000000ff5dd4 in boost::_bi::storage1<boost::_bi::value<kudu::Callback<void ()> > >::~storage1() (this=0x2f5d450, __in_chrg=<optimized out>) at /usr/include/boost/bind/storage.hpp:40
 #30 0x0000000000ff5dee in boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > >::~list1() (this=0x2f5d450, __in_chrg=<optimized out>) at /usr/include/boost/bind/bind.hpp:217
 #31 0x0000000000ff5e0c in boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > > >::~bind_t() (this=0x2f5d440, __in_chrg=<optimized out>) at /usr/include/boost/bind/bind.hpp:864
 #32 0x0000000000ff9f81 in boost::detail::function::functor_manager<boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > > > >::manager(boost::detail::function::function_buffer const&, boost::detail::function::function_buffer&, boost::detail::function::functor_manager_operation_type, mpl_::bool_<false>) (in_buffer=..., out_buffer=..., op=boost::detail::function::destroy_functor_tag) at /usr/include/boost/function/function_base.hpp:389
 #33 0x0000000000ff9aa3 in boost::detail::function::functor_manager<boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > > > >::manager(boost::detail::function::function_buffer const&, boost::detail::function::function_buffer&, boost::detail::function::functor_manager_operation_type, boost::detail::function::function_obj_tag) (in_buffer=..., out_buffer=..., op=boost::detail::function::destroy_functor_tag) at /usr/include/boost/function/function_base.hpp:412
 #34 0x0000000000ff92db in boost::detail::function::functor_manager<boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > > > >::manage(boost::detail::function::function_buffer const&, boost::detail::function::function_buffer&, boost::detail::function::functor_manager_operation_type) (in_buffer=..., out_buffer=..., op=boost::detail::function::destroy_functor_tag) at /usr/include/boost/function/function_base.hpp:440
 #35 0x0000000000d350bf in boost::detail::function::basic_vtable0<void>::clear (this=0x1c9d250 <void boost::function0<void>::assign_to<boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > > > >(boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > > >)::stored_vtable>, functor=...) at /usr/include/boost/function/function_template.hpp:504
 #36 0x0000000000d32caf in boost::function0<void>::clear (this=0x380f748) at /usr/include/boost/function/function_template.hpp:877
 #37 0x0000000000d30b00 in boost::function0<void>::~function0 (this=0x380f748, __in_chrg=<optimized out>) at /usr/include/boost/function/function_template.hpp:759
 #38 0x0000000000d2f7da in boost::function<void ()>::~function() (this=0x380f748, __in_chrg=<optimized out>) at /usr/include/boost/function/function_template.hpp:1048
 #39 0x0000000001aebba0 in kudu::FunctionRunnable::~FunctionRunnable (this=0x380f740, __in_chrg=<optimized out>) at src/kudu/util/threadpool.cc:27
 #40 0x0000000001aebbfc in kudu::FunctionRunnable::~FunctionRunnable (this=0x380f740, __in_chrg=<optimized out>) at src/kudu/util/threadpool.cc:27
 #41 0x0000000001aec69c in std::tr1::_Sp_deleter<kudu::FunctionRunnable>::operator() (this=0x30f5e38, __p=0x380f740) at /usr/include/c++/4.8/tr1/shared_ptr.h:285
 #42 0x0000000001aec635 in std::tr1::_Sp_counted_base_impl<kudu::FunctionRunnable*, std::tr1::_Sp_deleter<kudu::FunctionRunnable>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x30f5e20) at /usr/include/c++/4.8/tr1/shared_ptr.h:257
 #43 0x0000000000d147fc in std::tr1::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x30f5e20) at /usr/include/c++/4.8/tr1/shared_ptr.h:141
 #44 0x0000000000d10603 in std::tr1::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7fc907f00b48, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:341
 #45 0x00000000010724b4 in std::tr1::__shared_ptr<kudu::Runnable, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7fc907f00b40, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:541
 #46 0x00000000010724ce in std::tr1::shared_ptr<kudu::Runnable>::~shared_ptr (this=0x7fc907f00b40, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:985
 #47 0x0000000001aeaf76 in kudu::ThreadPool::QueueEntry::~QueueEntry (this=0x7fc907f00b40, __in_chrg=<optimized out>) at src/kudu/util/threadpool.h:159
 #48 0x0000000001aea7ae in kudu::ThreadPool::DispatchThread (this=0x36152c0, permanent=false) at src/kudu/util/threadpool.cc:278
 #49 0x0000000001aec560 in boost::_mfi::mf1<void, kudu::ThreadPool, bool>::operator() (this=0x30f5f60, p=0x36152c0, a1=false) at /usr/include/boost/bind/mem_fn_template.hpp:165
 #50 0x0000000001aec43b in boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> >::operator()<boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list0> (this=0x30f5f70, f=..., a=...) at /usr/include/boost/bind/bind.hpp:313
 #51 0x0000000001aec21b in boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >::operator() (this=0x30f5f60) at /usr/include/boost/bind/bind_template.hpp:20
 #52 0x0000000001aec14f in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >, void>::invoke (function_obj_ptr=...) at /usr/include/boost/function/function_template.hpp:153
 #53 0x00000000019dbb58 in boost::function0<void>::operator() (this=0x381c898) at /usr/include/boost/function/function_template.hpp:767
 #54 0x0000000001ae2440 in kudu::Thread::SuperviseThread (arg=0x381c870) at src/kudu/util/thread.cc:478
 #55 0x00007fc9288ae182 in start_thread (arg=0x7fc907f01700) at pthread_create.c:312
 #56 0x00007fc92779447d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. KUDU-690 Deadlock on shutdown when TS has multiple uncommitted CONFIG_CHANGE ops

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              mpercy Mike Percy
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: