[==========] Running 6 tests from 1 test case. [----------] Global test environment set-up. [----------] 6 tests from RaftConsensusTest [ RUN ] RaftConsensusTest.TestFollowersReplicateAndCommitMessage WARNING: Logging before InitGoogleLogging() is written to STDERR I0722 01:42:37.841500 6627 log.cc:251] Log is configured to *not* fsync() on all Append() calls I0722 01:42:37.855744 6627 raft_consensus.cc:66] T TestTablet P peer-0 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-0, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:37.858815 6627 raft_consensus.cc:66] T TestTablet P peer-1 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-1, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:37.858945 6627 raft_consensus.cc:66] T TestTablet P peer-2 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-2, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:37.859316 6627 raft_consensus.cc:80] T TestTablet P peer-0 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:37.859643 6627 raft_consensus.cc:80] T TestTablet P peer-1 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:37.861632 6627 raft_consensus.cc:168] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): replicating to peers... I0722 01:42:37.864364 6627 raft_consensus.cc:180] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): committing... I0722 01:42:37.865959 6627 raft_consensus.cc:203] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): a majority of peers have accepted the new configuration. Proceeding. I0722 01:42:37.866036 6627 raft_consensus.cc:80] T TestTablet P peer-2 [LEADER]: Started. Quorum: seqno: 1 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:37.872983 6627 raft_consensus.cc:510] T TestTablet P peer-0 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:37.873090 6627 raft_consensus_state.cc:226] T TestTablet P peer-0 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:37.873183 6627 raft_consensus_state.cc:244] T TestTablet P peer-0 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:37.873237 6627 raft_consensus_state.cc:248] T TestTablet P peer-0 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:37.873286 6627 raft_consensus.cc:521] T TestTablet P peer-0 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:37.873428 6627 raft_consensus.cc:510] T TestTablet P peer-1 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:37.873493 6627 raft_consensus_state.cc:226] T TestTablet P peer-1 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:37.873539 6627 raft_consensus_state.cc:244] T TestTablet P peer-1 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:37.873595 6627 raft_consensus_state.cc:248] T TestTablet P peer-1 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:37.873653 6627 raft_consensus.cc:521] T TestTablet P peer-1 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:37.873796 6627 raft_consensus.cc:510] T TestTablet P peer-2 [LEADER]: Raft consensus shutting down. I0722 01:42:37.874053 6627 consensus_peers.cc:411] Closing peer: peer-0 Waiting for outstanding requests to complete 0 I0722 01:42:37.874133 6627 consensus_peers.cc:411] Closing peer: peer-2 Waiting for outstanding requests to complete 0 I0722 01:42:37.874608 6627 consensus_peers.cc:411] Closing peer: peer-1 Waiting for outstanding requests to complete 0 I0722 01:42:37.874662 6627 raft_consensus_state.cc:226] T TestTablet P peer-2 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:37.874706 6627 raft_consensus_state.cc:244] T TestTablet P peer-2 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:37.874743 6627 raft_consensus_state.cc:248] T TestTablet P peer-2 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:37.874953 6627 raft_consensus.cc:521] T TestTablet P peer-2 [NON_PARTICIPANT]: Raft consensus Shutdown! [ OK ] RaftConsensusTest.TestFollowersReplicateAndCommitMessage (40 ms) [ RUN ] RaftConsensusTest.TestFollowersReplicateAndCommitSequence I0722 01:42:37.893465 6627 raft_consensus.cc:66] T TestTablet P peer-0 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-0, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:37.893633 6627 raft_consensus.cc:66] T TestTablet P peer-1 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-1, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:37.893823 6627 raft_consensus.cc:66] T TestTablet P peer-2 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-2, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:37.894038 6627 raft_consensus.cc:80] T TestTablet P peer-0 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:37.894232 6627 raft_consensus.cc:80] T TestTablet P peer-1 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:37.896446 6627 raft_consensus.cc:168] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): replicating to peers... I0722 01:42:37.898282 6627 raft_consensus.cc:180] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): committing... I0722 01:42:37.899341 6627 raft_consensus.cc:203] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): a majority of peers have accepted the new configuration. Proceeding. I0722 01:42:37.899410 6627 raft_consensus.cc:80] T TestTablet P peer-2 [LEADER]: Started. Quorum: seqno: 1 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:38.058207 6627 raft_consensus.cc:510] T TestTablet P peer-0 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:38.058310 6627 raft_consensus_state.cc:226] T TestTablet P peer-0 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.058379 6627 raft_consensus_state.cc:244] T TestTablet P peer-0 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.058426 6627 raft_consensus_state.cc:248] T TestTablet P peer-0 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.058470 6627 raft_consensus.cc:521] T TestTablet P peer-0 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:38.058675 6627 raft_consensus.cc:510] T TestTablet P peer-1 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:38.058780 6627 raft_consensus_state.cc:226] T TestTablet P peer-1 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.058857 6627 raft_consensus_state.cc:244] T TestTablet P peer-1 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.058909 6627 raft_consensus_state.cc:248] T TestTablet P peer-1 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.058982 6627 raft_consensus.cc:521] T TestTablet P peer-1 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:38.059128 6627 raft_consensus.cc:510] T TestTablet P peer-2 [LEADER]: Raft consensus shutting down. I0722 01:42:38.059365 6627 consensus_peers.cc:411] Closing peer: peer-0 Waiting for outstanding requests to complete 0 I0722 01:42:38.059418 6627 consensus_peers.cc:411] Closing peer: peer-2 Waiting for outstanding requests to complete 0 I0722 01:42:38.059633 6627 consensus_peers.cc:411] Closing peer: peer-1 Waiting for outstanding requests to complete 0 I0722 01:42:38.059715 6627 raft_consensus_state.cc:226] T TestTablet P peer-2 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.059774 6627 raft_consensus_state.cc:244] T TestTablet P peer-2 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.059823 6627 raft_consensus_state.cc:248] T TestTablet P peer-2 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.060060 6627 raft_consensus.cc:521] T TestTablet P peer-2 [NON_PARTICIPANT]: Raft consensus Shutdown! [ OK ] RaftConsensusTest.TestFollowersReplicateAndCommitSequence (186 ms) [ RUN ] RaftConsensusTest.TestConsensusContinuesIfAMinorityFallsBehind I0722 01:42:38.078359 6627 raft_consensus.cc:66] T TestTablet P peer-0 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-0, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.078547 6627 raft_consensus.cc:66] T TestTablet P peer-1 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-1, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.078678 6627 raft_consensus.cc:66] T TestTablet P peer-2 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-2, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.078876 6627 raft_consensus.cc:80] T TestTablet P peer-0 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:38.079093 6627 raft_consensus.cc:80] T TestTablet P peer-1 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:38.080529 6627 raft_consensus.cc:168] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): replicating to peers... I0722 01:42:38.081845 6627 raft_consensus.cc:180] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): committing... I0722 01:42:38.082660 6627 raft_consensus.cc:203] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): a majority of peers have accepted the new configuration. Proceeding. I0722 01:42:38.082912 6627 raft_consensus.cc:80] T TestTablet P peer-2 [LEADER]: Started. Quorum: seqno: 1 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6627) Cycle in lock order graph: M6751 (0x7d540000f278) => M6792 (0x7d0c00009320) => M6741 (0x7d540000f778) => M6751 Mutex M6792 acquired here while holding mutex M6751 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:199 (raft_consensus-test+0x0000001392a9) #5 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #6 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #7 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #8 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #9 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #10 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #11 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #12 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #13 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:426 (raft_consensus-test+0x00000010ca83) #14 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #15 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M6751 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForConfigChange(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:123 (libconsensus.so+0x00000008a062) #6 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:91 (libconsensus.so+0x00000007cc84) #7 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #8 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #9 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #10 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:426 (raft_consensus-test+0x00000010ca83) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #12 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M6741 acquired here while holding mutex M6792 in thread T39: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForUpdate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:134 (libconsensus.so+0x00000008a17e) #6 kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusStatusPB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:405 (libconsensus.so+0x00000007fb84) #7 kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:375 (libconsensus.so+0x00000007f746) #8 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:219 (raft_consensus-test+0x000000139645) #9 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #10 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #11 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #13 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #14 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #15 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #16 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #17 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #18 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #19 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #20 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #21 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #22 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #23 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #24 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #25 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #26 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M6792 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:217 (raft_consensus-test+0x000000139607) #5 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #6 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #7 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #8 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #9 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #10 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #11 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #12 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #13 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #14 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #16 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #17 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #18 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #19 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #20 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #21 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #22 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M6751 acquired here while holding mutex M6741 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForReplicate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:100 (libconsensus.so+0x000000089e7e) #6 kudu::consensus::RaftConsensus::Replicate(kudu::consensus::ConsensusRound*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:214 (libconsensus.so+0x00000007e5d6) #7 kudu::consensus::RaftConsensusTest::AppendDummyMessage(kudu::consensus::RaftConsensus*, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:177 (raft_consensus-test+0x00000010feb6) #8 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:441 (raft_consensus-test+0x00000010cbf4) #9 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #10 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M6741 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForRead(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:95 (libconsensus.so+0x000000089df6) #6 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:436 (raft_consensus-test+0x00000010cb3f) #7 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #8 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Thread T9 'noop-peer-pool ' (tid=6668, running) created by main thread at: #0 pthread_create :0 (raft_consensus-test+0x0000000a7511) #1 boost::thread::start_thread_noexcept() :0 (libboost_thread.so.1.54.0+0x00000000b3b8) #2 thread, long *), kudu::Thread *, boost::function, long *> /usr/include/boost/thread/detail/thread.hpp:404 (libkudu_util.so+0x00000014dd83) #3 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.cc:403 (libkudu_util.so+0x00000014bb04) #4 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.h:133 (libkudu_util.so+0x000000159fcb) #5 kudu::ThreadPool::CreateThreadUnlocked() /home/todd/git/kudu/src/util/threadpool.cc:291 (libkudu_util.so+0x0000001585a1) #6 kudu::ThreadPool::Submit(std::tr1::shared_ptr const&) /home/todd/git/kudu/src/util/threadpool.cc:168 (libkudu_util.so+0x000000158a0a) #7 kudu::ThreadPool::SubmitFunc(boost::function const&) /home/todd/git/kudu/src/util/threadpool.cc:141 (libkudu_util.so+0x000000158815) #8 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:201 (raft_consensus-test+0x000000139312) #9 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #10 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #11 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #12 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #13 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #14 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #15 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #16 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #17 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:426 (raft_consensus-test+0x00000010ca83) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #19 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 AnnotateRWLockAcquired ================== ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6627) Cycle in lock order graph: M6760 (0x7d3c0000d968) => M6792 (0x7d0c00009320) => M6741 (0x7d540000f778) => M6760 Mutex M6792 acquired here while holding mutex M6760 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:199 (raft_consensus-test+0x0000001392a9) #5 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #6 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #7 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #8 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #9 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #10 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #11 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #12 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #13 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:426 (raft_consensus-test+0x00000010ca83) #14 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #15 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M6760 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:334 (libconsensus.so+0x00000006d4a2) #5 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #6 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #7 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #8 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #9 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #10 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #11 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:426 (raft_consensus-test+0x00000010ca83) #12 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #13 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M6741 acquired here while holding mutex M6792 in thread T39: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForUpdate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:134 (libconsensus.so+0x00000008a17e) #6 kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusStatusPB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:405 (libconsensus.so+0x00000007fb84) #7 kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:375 (libconsensus.so+0x00000007f746) #8 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:219 (raft_consensus-test+0x000000139645) #9 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #10 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #11 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #13 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #14 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #15 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #16 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #17 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #18 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #19 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #20 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #21 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #22 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #23 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #24 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #25 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #26 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M6792 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:217 (raft_consensus-test+0x000000139607) #5 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #6 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #7 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #8 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #9 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #10 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #11 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #12 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #13 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #14 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #16 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #17 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #18 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #19 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #20 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #21 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #22 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M6760 acquired here while holding mutex M6741 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:334 (libconsensus.so+0x00000006d4a2) #5 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #6 kudu::consensus::RaftConsensus::Replicate(kudu::consensus::ConsensusRound*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:249 (libconsensus.so+0x00000007e7f2) #7 kudu::consensus::RaftConsensusTest::AppendDummyMessage(kudu::consensus::RaftConsensus*, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:177 (raft_consensus-test+0x00000010feb6) #8 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:441 (raft_consensus-test+0x00000010cbf4) #9 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #10 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M6741 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForRead(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:95 (libconsensus.so+0x000000089df6) #6 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:436 (raft_consensus-test+0x00000010cb3f) #7 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #8 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Thread T9 'noop-peer-pool ' (tid=6668, running) created by main thread at: #0 pthread_create :0 (raft_consensus-test+0x0000000a7511) #1 boost::thread::start_thread_noexcept() :0 (libboost_thread.so.1.54.0+0x00000000b3b8) #2 thread, long *), kudu::Thread *, boost::function, long *> /usr/include/boost/thread/detail/thread.hpp:404 (libkudu_util.so+0x00000014dd83) #3 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.cc:403 (libkudu_util.so+0x00000014bb04) #4 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.h:133 (libkudu_util.so+0x000000159fcb) #5 kudu::ThreadPool::CreateThreadUnlocked() /home/todd/git/kudu/src/util/threadpool.cc:291 (libkudu_util.so+0x0000001585a1) #6 kudu::ThreadPool::Submit(std::tr1::shared_ptr const&) /home/todd/git/kudu/src/util/threadpool.cc:168 (libkudu_util.so+0x000000158a0a) #7 kudu::ThreadPool::SubmitFunc(boost::function const&) /home/todd/git/kudu/src/util/threadpool.cc:141 (libkudu_util.so+0x000000158815) #8 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:201 (raft_consensus-test+0x000000139312) #9 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #10 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #11 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #12 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #13 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #14 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #15 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #16 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #17 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:426 (raft_consensus-test+0x00000010ca83) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #19 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 AnnotateRWLockAcquired ================== ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6627) Cycle in lock order graph: M6792 (0x7d0c00009320) => M6741 (0x7d540000f778) => M6792 Mutex M6741 acquired here while holding mutex M6792 in thread T39: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForUpdate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:134 (libconsensus.so+0x00000008a17e) #6 kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusStatusPB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:405 (libconsensus.so+0x00000007fb84) #7 kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:375 (libconsensus.so+0x00000007f746) #8 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:219 (raft_consensus-test+0x000000139645) #9 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #10 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #11 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #13 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #14 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #15 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #16 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #17 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #18 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #19 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #20 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #21 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #22 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #23 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #24 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #25 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #26 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M6792 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:217 (raft_consensus-test+0x000000139607) #5 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #6 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #7 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #8 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #9 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #10 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #11 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #12 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #13 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #14 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #16 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #17 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #18 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #19 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #20 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #21 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #22 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M6792 acquired here while holding mutex M6741 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:199 (raft_consensus-test+0x0000001392a9) #5 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #6 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #7 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #8 kudu::consensus::RaftConsensus::Replicate(kudu::consensus::ConsensusRound*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:249 (libconsensus.so+0x00000007e7f2) #9 kudu::consensus::RaftConsensusTest::AppendDummyMessage(kudu::consensus::RaftConsensus*, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:177 (raft_consensus-test+0x00000010feb6) #10 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:441 (raft_consensus-test+0x00000010cbf4) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #12 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M6741 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForRead(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:95 (libconsensus.so+0x000000089df6) #6 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:436 (raft_consensus-test+0x00000010cb3f) #7 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #8 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Thread T9 'noop-peer-pool ' (tid=6668, running) created by main thread at: #0 pthread_create :0 (raft_consensus-test+0x0000000a7511) #1 boost::thread::start_thread_noexcept() :0 (libboost_thread.so.1.54.0+0x00000000b3b8) #2 thread, long *), kudu::Thread *, boost::function, long *> /usr/include/boost/thread/detail/thread.hpp:404 (libkudu_util.so+0x00000014dd83) #3 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.cc:403 (libkudu_util.so+0x00000014bb04) #4 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.h:133 (libkudu_util.so+0x000000159fcb) #5 kudu::ThreadPool::CreateThreadUnlocked() /home/todd/git/kudu/src/util/threadpool.cc:291 (libkudu_util.so+0x0000001585a1) #6 kudu::ThreadPool::Submit(std::tr1::shared_ptr const&) /home/todd/git/kudu/src/util/threadpool.cc:168 (libkudu_util.so+0x000000158a0a) #7 kudu::ThreadPool::SubmitFunc(boost::function const&) /home/todd/git/kudu/src/util/threadpool.cc:141 (libkudu_util.so+0x000000158815) #8 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:201 (raft_consensus-test+0x000000139312) #9 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #10 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #11 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #12 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #13 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #14 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #15 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #16 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #17 kudu::consensus::RaftConsensusTest_TestConsensusContinuesIfAMinorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:426 (raft_consensus-test+0x00000010ca83) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #19 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 AnnotateRWLockAcquired ================== I0722 01:42:38.358935 6627 raft_consensus.cc:510] T TestTablet P peer-0 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:38.359026 6627 raft_consensus_state.cc:226] T TestTablet P peer-0 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.359071 6627 raft_consensus_state.cc:244] T TestTablet P peer-0 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.359128 6627 raft_consensus_state.cc:248] T TestTablet P peer-0 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.359204 6627 raft_consensus.cc:521] T TestTablet P peer-0 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:38.359387 6627 raft_consensus.cc:510] T TestTablet P peer-1 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:38.359498 6627 raft_consensus_state.cc:226] T TestTablet P peer-1 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.359554 6627 raft_consensus_state.cc:244] T TestTablet P peer-1 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.359607 6627 raft_consensus_state.cc:248] T TestTablet P peer-1 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.359661 6627 raft_consensus.cc:521] T TestTablet P peer-1 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:38.359839 6627 raft_consensus.cc:510] T TestTablet P peer-2 [LEADER]: Raft consensus shutting down. I0722 01:42:38.360226 6627 consensus_peers.cc:411] Closing peer: peer-0 Waiting for outstanding requests to complete 0 I0722 01:42:38.360308 6627 consensus_peers.cc:411] Closing peer: peer-2 Waiting for outstanding requests to complete 0 I0722 01:42:38.360563 6627 consensus_peers.cc:411] Closing peer: peer-1 Waiting for outstanding requests to complete 0 I0722 01:42:38.360644 6627 raft_consensus_state.cc:226] T TestTablet P peer-2 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.360702 6627 raft_consensus_state.cc:244] T TestTablet P peer-2 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.360774 6627 raft_consensus_state.cc:248] T TestTablet P peer-2 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.361006 6627 raft_consensus.cc:521] T TestTablet P peer-2 [NON_PARTICIPANT]: Raft consensus Shutdown! [ OK ] RaftConsensusTest.TestConsensusContinuesIfAMinorityFallsBehind (300 ms) [ RUN ] RaftConsensusTest.TestConsensusStopsIfAMajorityFallsBehind I0722 01:42:38.381177 6627 raft_consensus.cc:66] T TestTablet P peer-0 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-0, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.381337 6627 raft_consensus.cc:66] T TestTablet P peer-1 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-1, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.381494 6627 raft_consensus.cc:66] T TestTablet P peer-2 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-2, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.381675 6627 raft_consensus.cc:80] T TestTablet P peer-0 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:38.381865 6627 raft_consensus.cc:80] T TestTablet P peer-1 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:38.383157 6627 raft_consensus.cc:168] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): replicating to peers... I0722 01:42:38.384421 6627 raft_consensus.cc:180] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): committing... I0722 01:42:38.385135 6627 raft_consensus.cc:203] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): a majority of peers have accepted the new configuration. Proceeding. I0722 01:42:38.385202 6627 raft_consensus.cc:80] T TestTablet P peer-2 [LEADER]: Started. Quorum: seqno: 1 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6627) Cycle in lock order graph: M7599 (0x7d540000f778) => M7640 (0x7d0c00022f70) => M7589 (0x7d540000f278) => M7599 Mutex M7640 acquired here while holding mutex M7599 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:199 (raft_consensus-test+0x0000001392a9) #5 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #6 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #7 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #8 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #9 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #10 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #11 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #12 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #13 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #14 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #15 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7599 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForConfigChange(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:123 (libconsensus.so+0x00000008a062) #6 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:91 (libconsensus.so+0x00000007cc84) #7 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #8 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #9 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #10 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #12 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7589 acquired here while holding mutex M7640 in thread T54: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForUpdate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:134 (libconsensus.so+0x00000008a17e) #6 kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusStatusPB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:405 (libconsensus.so+0x00000007fb84) #7 kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:375 (libconsensus.so+0x00000007f746) #8 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:219 (raft_consensus-test+0x000000139645) #9 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #10 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #11 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #13 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #14 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #15 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #16 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #17 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #18 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #19 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #20 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #21 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #22 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #23 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #24 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #25 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #26 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7640 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:217 (raft_consensus-test+0x000000139607) #5 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #6 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #7 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #8 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #9 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #10 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #11 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #12 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #13 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #14 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #16 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #17 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #18 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #19 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #20 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #21 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #22 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7599 acquired here while holding mutex M7589 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForReplicate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:100 (libconsensus.so+0x000000089e7e) #6 kudu::consensus::RaftConsensus::Replicate(kudu::consensus::ConsensusRound*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:214 (libconsensus.so+0x00000007e5d6) #7 kudu::consensus::RaftConsensusTest::AppendDummyMessage(kudu::consensus::RaftConsensus*, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:177 (raft_consensus-test+0x00000010feb6) #8 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:479 (raft_consensus-test+0x00000010d3cf) #9 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #10 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7589 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForRead(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:95 (libconsensus.so+0x000000089df6) #6 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:472 (raft_consensus-test+0x00000010d288) #7 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #8 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Thread T30 'noop-peer-pool ' (tid=6689, running) created by main thread at: #0 pthread_create :0 (raft_consensus-test+0x0000000a7511) #1 boost::thread::start_thread_noexcept() :0 (libboost_thread.so.1.54.0+0x00000000b3b8) #2 thread, long *), kudu::Thread *, boost::function, long *> /usr/include/boost/thread/detail/thread.hpp:404 (libkudu_util.so+0x00000014dd83) #3 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.cc:403 (libkudu_util.so+0x00000014bb04) #4 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.h:133 (libkudu_util.so+0x000000159fcb) #5 kudu::ThreadPool::CreateThreadUnlocked() /home/todd/git/kudu/src/util/threadpool.cc:291 (libkudu_util.so+0x0000001585a1) #6 kudu::ThreadPool::Submit(std::tr1::shared_ptr const&) /home/todd/git/kudu/src/util/threadpool.cc:168 (libkudu_util.so+0x000000158a0a) #7 kudu::ThreadPool::SubmitFunc(boost::function const&) /home/todd/git/kudu/src/util/threadpool.cc:141 (libkudu_util.so+0x000000158815) #8 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:201 (raft_consensus-test+0x000000139312) #9 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #10 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #11 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #12 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #13 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #14 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #15 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #16 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #17 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #19 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 AnnotateRWLockAcquired ================== ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6627) Cycle in lock order graph: M7608 (0x7d3c0000db48) => M7640 (0x7d0c00022f70) => M7589 (0x7d540000f278) => M7608 Mutex M7640 acquired here while holding mutex M7608 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:199 (raft_consensus-test+0x0000001392a9) #5 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #6 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #7 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #8 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #9 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #10 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #11 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #12 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #13 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #14 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #15 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7608 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:334 (libconsensus.so+0x00000006d4a2) #5 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #6 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #7 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #8 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #9 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #10 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #11 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #12 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #13 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7589 acquired here while holding mutex M7640 in thread T54: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForUpdate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:134 (libconsensus.so+0x00000008a17e) #6 kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusStatusPB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:405 (libconsensus.so+0x00000007fb84) #7 kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:375 (libconsensus.so+0x00000007f746) #8 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:219 (raft_consensus-test+0x000000139645) #9 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #10 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #11 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #13 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #14 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #15 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #16 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #17 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #18 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #19 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #20 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #21 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #22 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #23 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #24 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #25 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #26 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7640 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:217 (raft_consensus-test+0x000000139607) #5 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #6 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #7 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #8 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #9 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #10 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #11 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #12 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #13 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #14 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #16 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #17 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #18 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #19 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #20 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #21 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #22 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7608 acquired here while holding mutex M7589 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:334 (libconsensus.so+0x00000006d4a2) #5 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #6 kudu::consensus::RaftConsensus::Replicate(kudu::consensus::ConsensusRound*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:249 (libconsensus.so+0x00000007e7f2) #7 kudu::consensus::RaftConsensusTest::AppendDummyMessage(kudu::consensus::RaftConsensus*, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:177 (raft_consensus-test+0x00000010feb6) #8 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:479 (raft_consensus-test+0x00000010d3cf) #9 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #10 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7589 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForRead(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:95 (libconsensus.so+0x000000089df6) #6 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:472 (raft_consensus-test+0x00000010d288) #7 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #8 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Thread T30 'noop-peer-pool ' (tid=6689, running) created by main thread at: #0 pthread_create :0 (raft_consensus-test+0x0000000a7511) #1 boost::thread::start_thread_noexcept() :0 (libboost_thread.so.1.54.0+0x00000000b3b8) #2 thread, long *), kudu::Thread *, boost::function, long *> /usr/include/boost/thread/detail/thread.hpp:404 (libkudu_util.so+0x00000014dd83) #3 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.cc:403 (libkudu_util.so+0x00000014bb04) #4 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.h:133 (libkudu_util.so+0x000000159fcb) #5 kudu::ThreadPool::CreateThreadUnlocked() /home/todd/git/kudu/src/util/threadpool.cc:291 (libkudu_util.so+0x0000001585a1) #6 kudu::ThreadPool::Submit(std::tr1::shared_ptr const&) /home/todd/git/kudu/src/util/threadpool.cc:168 (libkudu_util.so+0x000000158a0a) #7 kudu::ThreadPool::SubmitFunc(boost::function const&) /home/todd/git/kudu/src/util/threadpool.cc:141 (libkudu_util.so+0x000000158815) #8 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:201 (raft_consensus-test+0x000000139312) #9 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #10 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #11 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #12 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #13 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #14 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #15 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #16 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #17 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #19 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 AnnotateRWLockAcquired ================== ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6627) Cycle in lock order graph: M7640 (0x7d0c00022f70) => M7589 (0x7d540000f278) => M7640 Mutex M7589 acquired here while holding mutex M7640 in thread T54: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForUpdate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:134 (libconsensus.so+0x00000008a17e) #6 kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusStatusPB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:405 (libconsensus.so+0x00000007fb84) #7 kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:375 (libconsensus.so+0x00000007f746) #8 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:219 (raft_consensus-test+0x000000139645) #9 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #10 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #11 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #13 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #14 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #15 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #16 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #17 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #18 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #19 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #20 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #21 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #22 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #23 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #24 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #25 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #26 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7640 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:217 (raft_consensus-test+0x000000139607) #5 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #6 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #7 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #8 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #9 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #10 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #11 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #12 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #13 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #14 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #16 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #17 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #18 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #19 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #20 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #21 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #22 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7640 acquired here while holding mutex M7589 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:199 (raft_consensus-test+0x0000001392a9) #5 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #6 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #7 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #8 kudu::consensus::RaftConsensus::Replicate(kudu::consensus::ConsensusRound*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:249 (libconsensus.so+0x00000007e7f2) #9 kudu::consensus::RaftConsensusTest::AppendDummyMessage(kudu::consensus::RaftConsensus*, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:177 (raft_consensus-test+0x00000010feb6) #10 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:479 (raft_consensus-test+0x00000010d3cf) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #12 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7589 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForRead(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:95 (libconsensus.so+0x000000089df6) #6 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:472 (raft_consensus-test+0x00000010d288) #7 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #8 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Thread T30 'noop-peer-pool ' (tid=6689, running) created by main thread at: #0 pthread_create :0 (raft_consensus-test+0x0000000a7511) #1 boost::thread::start_thread_noexcept() :0 (libboost_thread.so.1.54.0+0x00000000b3b8) #2 thread, long *), kudu::Thread *, boost::function, long *> /usr/include/boost/thread/detail/thread.hpp:404 (libkudu_util.so+0x00000014dd83) #3 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.cc:403 (libkudu_util.so+0x00000014bb04) #4 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.h:133 (libkudu_util.so+0x000000159fcb) #5 kudu::ThreadPool::CreateThreadUnlocked() /home/todd/git/kudu/src/util/threadpool.cc:291 (libkudu_util.so+0x0000001585a1) #6 kudu::ThreadPool::Submit(std::tr1::shared_ptr const&) /home/todd/git/kudu/src/util/threadpool.cc:168 (libkudu_util.so+0x000000158a0a) #7 kudu::ThreadPool::SubmitFunc(boost::function const&) /home/todd/git/kudu/src/util/threadpool.cc:141 (libkudu_util.so+0x000000158815) #8 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:201 (raft_consensus-test+0x000000139312) #9 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #10 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #11 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #12 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #13 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #14 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #15 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #16 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #17 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #19 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 AnnotateRWLockAcquired ================== ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6627) Cycle in lock order graph: M7625 (0x7d3c0000da58) => M7655 (0x7d0c00029750) => M7594 (0x7d540000f4f8) => M7625 Mutex M7655 acquired here while holding mutex M7625 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:199 (raft_consensus-test+0x0000001392a9) #5 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #6 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #7 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #8 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #9 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #10 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #11 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #12 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #13 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #14 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #15 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7625 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:334 (libconsensus.so+0x00000006d4a2) #5 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #6 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #7 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #8 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #9 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #10 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #11 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #12 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #13 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7594 acquired here while holding mutex M7655 in thread T55: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForUpdate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:134 (libconsensus.so+0x00000008a17e) #6 kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusStatusPB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:405 (libconsensus.so+0x00000007fb84) #7 kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:375 (libconsensus.so+0x00000007f746) #8 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:219 (raft_consensus-test+0x000000139645) #9 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #10 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #11 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #13 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #14 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #15 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #16 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #17 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #18 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #19 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #20 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #21 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #22 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #23 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #24 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #25 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #26 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7655 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:217 (raft_consensus-test+0x000000139607) #5 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #6 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #7 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #8 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #9 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #10 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #11 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #12 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #13 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #14 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #16 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #17 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #18 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #19 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #20 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #21 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #22 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7625 acquired here while holding mutex M7594 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:334 (libconsensus.so+0x00000006d4a2) #5 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #6 kudu::consensus::RaftConsensus::Replicate(kudu::consensus::ConsensusRound*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:249 (libconsensus.so+0x00000007e7f2) #7 kudu::consensus::RaftConsensusTest::AppendDummyMessage(kudu::consensus::RaftConsensus*, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:177 (raft_consensus-test+0x00000010feb6) #8 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:479 (raft_consensus-test+0x00000010d3cf) #9 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #10 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7594 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForRead(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:95 (libconsensus.so+0x000000089df6) #6 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:476 (raft_consensus-test+0x00000010d339) #7 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #8 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Thread T23 'noop-peer-pool ' (tid=6690, running) created by main thread at: #0 pthread_create :0 (raft_consensus-test+0x0000000a7511) #1 boost::thread::start_thread_noexcept() :0 (libboost_thread.so.1.54.0+0x00000000b3b8) #2 thread, long *), kudu::Thread *, boost::function, long *> /usr/include/boost/thread/detail/thread.hpp:404 (libkudu_util.so+0x00000014dd83) #3 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.cc:403 (libkudu_util.so+0x00000014bb04) #4 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.h:133 (libkudu_util.so+0x000000159fcb) #5 kudu::ThreadPool::CreateThreadUnlocked() /home/todd/git/kudu/src/util/threadpool.cc:291 (libkudu_util.so+0x0000001585a1) #6 kudu::ThreadPool::Submit(std::tr1::shared_ptr const&) /home/todd/git/kudu/src/util/threadpool.cc:168 (libkudu_util.so+0x000000158a0a) #7 kudu::ThreadPool::SubmitFunc(boost::function const&) /home/todd/git/kudu/src/util/threadpool.cc:141 (libkudu_util.so+0x000000158815) #8 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:201 (raft_consensus-test+0x000000139312) #9 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #10 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #11 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #12 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #13 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #14 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #15 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #16 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #17 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #19 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 AnnotateRWLockAcquired ================== ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6627) Cycle in lock order graph: M7655 (0x7d0c00029750) => M7594 (0x7d540000f4f8) => M7655 Mutex M7594 acquired here while holding mutex M7655 in thread T55: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForUpdate(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:134 (libconsensus.so+0x00000008a17e) #6 kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusStatusPB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:405 (libconsensus.so+0x00000007fb84) #7 kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:375 (libconsensus.so+0x00000007f746) #8 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:219 (raft_consensus-test+0x000000139645) #9 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #10 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #11 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #13 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #14 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #15 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #16 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #17 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #18 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #19 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #20 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #21 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #22 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #23 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #24 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #25 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #26 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7655 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::SendToOtherPeer(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/todd/git/kudu/src/consensus/consensus-test-util.h:217 (raft_consensus-test+0x000000139607) #5 boost::_mfi::mf2::operator()(kudu::consensus::LocalTestPeerProxy*, kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (raft_consensus-test+0x00000013a11e) #6 void boost::_bi::list3, boost::_bi::value, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (raft_consensus-test+0x00000013a07b) #7 boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (raft_consensus-test+0x000000139fe3) #8 boost::detail::function::void_function_obj_invoker0, boost::_bi::list3, boost::_bi::value, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (raft_consensus-test+0x000000139dd1) #9 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #10 kudu::FunctionRunnable::Run() /home/todd/git/kudu/src/util/threadpool.cc:33 (libkudu_util.so+0x00000015b66d) #11 kudu::ThreadPool::DispatchThread(bool) /home/todd/git/kudu/src/util/threadpool.cc:265 (libkudu_util.so+0x000000158f66) #12 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:165 (libkudu_util.so+0x00000015a595) #13 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:313 (libkudu_util.so+0x00000015a4fb) #14 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000015a483) #15 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x00000015a281) #16 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:766 (raft_consensus-test+0x000000132571) #17 kudu::Thread::SuperviseThread(boost::function, long*) /home/todd/git/kudu/src/util/thread.cc:464 (libkudu_util.so+0x00000014c086) #18 boost::_mfi::mf2, long*>::operator()(kudu::Thread*, boost::function, long*) const /usr/include/boost/bind/mem_fn_template.hpp:280 (libkudu_util.so+0x000000150010) #19 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value >::operator(), long*>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2, long*>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:392 (libkudu_util.so+0x00000014ff2f) #20 boost::_bi::bind_t, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x00000014fe93) #21 boost::detail::thread_data, long*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::run() /usr/include/boost/thread/detail/thread.hpp:117 (libkudu_util.so+0x00000014f150) #22 boost::this_thread::interruption_requested() :0 (libboost_thread.so.1.54.0+0x00000000ba49) Mutex M7655 acquired here while holding mutex M7594 in main thread: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 lock_guard /usr/include/boost/thread/lock_guard.hpp:38 (raft_consensus-test+0x00000012e618) #4 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:199 (raft_consensus-test+0x0000001392a9) #5 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #6 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #7 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #8 kudu::consensus::RaftConsensus::Replicate(kudu::consensus::ConsensusRound*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:249 (libconsensus.so+0x00000007e7f2) #9 kudu::consensus::RaftConsensusTest::AppendDummyMessage(kudu::consensus::RaftConsensus*, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:177 (raft_consensus-test+0x00000010feb6) #10 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:479 (raft_consensus-test+0x00000010d3cf) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #12 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Mutex M7594 previously acquired by the same thread here: #0 AnnotateRWLockAcquired :0 (raft_consensus-test+0x0000000ed053) #1 base::SpinLock::Lock() /home/todd/git/kudu/src/gutil/spinlock.h:77 (raft_consensus-test+0x00000012e6d5) #2 kudu::simple_spinlock::lock() /home/todd/git/kudu/src/util/locks.h:29 (raft_consensus-test+0x00000012e679) #3 boost::unique_lock::lock() /usr/include/boost/thread/lock_types.hpp:346 (libconsensus.so+0x00000009148a) #4 unique_lock /usr/include/boost/thread/lock_types.hpp:124 (libconsensus.so+0x00000008d6a5) #5 kudu::consensus::ReplicaState::LockForRead(boost::unique_lock*) /home/todd/git/kudu/src/consensus/raft_consensus_state.cc:95 (libconsensus.so+0x000000089df6) #6 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:476 (raft_consensus-test+0x00000010d339) #7 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #8 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) Thread T23 'noop-peer-pool ' (tid=6690, running) created by main thread at: #0 pthread_create :0 (raft_consensus-test+0x0000000a7511) #1 boost::thread::start_thread_noexcept() :0 (libboost_thread.so.1.54.0+0x00000000b3b8) #2 thread, long *), kudu::Thread *, boost::function, long *> /usr/include/boost/thread/detail/thread.hpp:404 (libkudu_util.so+0x00000014dd83) #3 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.cc:403 (libkudu_util.so+0x00000014bb04) #4 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr*) /home/todd/git/kudu/src/util/thread.h:133 (libkudu_util.so+0x000000159fcb) #5 kudu::ThreadPool::CreateThreadUnlocked() /home/todd/git/kudu/src/util/threadpool.cc:291 (libkudu_util.so+0x0000001585a1) #6 kudu::ThreadPool::Submit(std::tr1::shared_ptr const&) /home/todd/git/kudu/src/util/threadpool.cc:168 (libkudu_util.so+0x000000158a0a) #7 kudu::ThreadPool::SubmitFunc(boost::function const&) /home/todd/git/kudu/src/util/threadpool.cc:141 (libkudu_util.so+0x000000158815) #8 kudu::consensus::LocalTestPeerProxy::UpdateAsync(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcController*, boost::function const&) /home/todd/git/kudu/src/consensus/consensus-test-util.h:201 (raft_consensus-test+0x000000139312) #9 kudu::consensus::RemotePeer::ProcessNextRequest() /home/todd/git/kudu/src/consensus/consensus_peers.cc:209 (libconsensus.so+0x0000000714b6) #10 kudu::consensus::Peer::SignalRequest(bool) /home/todd/git/kudu/src/consensus/consensus_peers.cc:358 (libconsensus.so+0x00000006d5bc) #11 kudu::consensus::RaftConsensus::SignalRequestToPeers(bool) /home/todd/git/kudu/src/consensus/raft_consensus.cc:492 (libconsensus.so+0x00000007e4a2) #12 kudu::consensus::RaftConsensus::PushConfigurationToPeersUnlocked() /home/todd/git/kudu/src/consensus/raft_consensus.cc:177 (libconsensus.so+0x00000007dd31) #13 kudu::consensus::RaftConsensus::ChangeConfig(kudu::metadata::QuorumPB) /home/todd/git/kudu/src/consensus/raft_consensus.cc:97 (libconsensus.so+0x00000007cd3a) #14 kudu::consensus::RaftConsensus::Start(kudu::metadata::QuorumPB const&, kudu::consensus::ConsensusBootstrapInfo const&, gscoped_ptr >*) /home/todd/git/kudu/src/consensus/raft_consensus.cc:75 (libconsensus.so+0x00000007c8eb) #15 kudu::consensus::RaftConsensusTest::StartPeers() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:131 (raft_consensus-test+0x0000001122c5) #16 kudu::consensus::RaftConsensusTest::BuildAndStartQuorum(int) /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:146 (raft_consensus-test+0x00000010fbc8) #17 kudu::consensus::RaftConsensusTest_TestConsensusStopsIfAMajorityFallsBehind_Test::TestBody() /home/todd/git/kudu/src/consensus/raft_consensus-test.cc:462 (raft_consensus-test+0x00000010d1e1) #18 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) :0 (libkudu_test_main.so+0x0000000639ed) #19 __libc_start_main /build/buildd/eglibc-2.17/csu/libc-start.c:260 (libc.so.6+0x000000021de4) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 AnnotateRWLockAcquired ================== I0722 01:42:38.960614 6627 raft_consensus.cc:510] T TestTablet P peer-0 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:38.960708 6627 raft_consensus_state.cc:226] T TestTablet P peer-0 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.960760 6627 raft_consensus_state.cc:244] T TestTablet P peer-0 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.960801 6627 raft_consensus_state.cc:248] T TestTablet P peer-0 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.960873 6627 raft_consensus.cc:521] T TestTablet P peer-0 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:38.961026 6627 raft_consensus.cc:510] T TestTablet P peer-1 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:38.961087 6627 raft_consensus_state.cc:226] T TestTablet P peer-1 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.961135 6627 raft_consensus_state.cc:244] T TestTablet P peer-1 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.961187 6627 raft_consensus_state.cc:248] T TestTablet P peer-1 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.961246 6627 raft_consensus.cc:521] T TestTablet P peer-1 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:38.961395 6627 raft_consensus.cc:510] T TestTablet P peer-2 [LEADER]: Raft consensus shutting down. I0722 01:42:38.961650 6627 consensus_peers.cc:411] Closing peer: peer-0 Waiting for outstanding requests to complete 0 I0722 01:42:38.961725 6627 consensus_peers.cc:411] Closing peer: peer-2 Waiting for outstanding requests to complete 0 I0722 01:42:38.962082 6627 consensus_peers.cc:411] Closing peer: peer-1 Waiting for outstanding requests to complete 0 I0722 01:42:38.962158 6627 raft_consensus_state.cc:226] T TestTablet P peer-2 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:38.962225 6627 raft_consensus_state.cc:244] T TestTablet P peer-2 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:38.962287 6627 raft_consensus_state.cc:248] T TestTablet P peer-2 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:38.962607 6627 raft_consensus.cc:521] T TestTablet P peer-2 [NON_PARTICIPANT]: Raft consensus Shutdown! [ OK ] RaftConsensusTest.TestConsensusStopsIfAMajorityFallsBehind (601 ms) [ RUN ] RaftConsensusTest.TestReplicasHandleCommunicationErrors I0722 01:42:38.979678 6627 raft_consensus.cc:66] T TestTablet P peer-0 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-0, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.979888 6627 raft_consensus.cc:66] T TestTablet P peer-1 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-1, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.980031 6627 raft_consensus.cc:66] T TestTablet P peer-2 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-2, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:38.980223 6627 raft_consensus.cc:80] T TestTablet P peer-0 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:38.980376 6627 raft_consensus.cc:80] T TestTablet P peer-1 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:38.981616 6627 raft_consensus.cc:168] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): replicating to peers... I0722 01:42:38.982796 6627 raft_consensus.cc:180] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): committing... I0722 01:42:38.983561 6627 raft_consensus.cc:203] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): a majority of peers have accepted the new configuration. Proceeding. I0722 01:42:38.983626 6627 raft_consensus.cc:80] T TestTablet P peer-2 [LEADER]: Started. Quorum: seqno: 1 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } W0722 01:42:38.985175 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.985296 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.985718 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 2 times. W0722 01:42:38.985872 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 2 times. W0722 01:42:38.986300 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 3 times. W0722 01:42:38.987278 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.987428 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.988225 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.989187 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.989774 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.990629 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.991294 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.992069 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.992805 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.992933 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.993811 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.994725 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.995411 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.996147 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.996911 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.997066 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.997797 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.998739 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:38.999677 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.000229 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.001114 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.001747 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.002460 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.003074 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.003984 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.004398 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.005429 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.006019 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.006911 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.007652 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.008453 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.009091 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.010365 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.010905 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.011765 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.012358 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.013141 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.013731 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.014513 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.015113 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.015923 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.016525 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.017590 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.018622 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.019361 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 2 times. W0722 01:42:39.020328 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.020665 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.022589 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.023166 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.024016 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.025270 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.026471 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.026517 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.028201 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.028667 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.029745 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.030267 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 2 times. W0722 01:42:39.031292 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 3 times. W0722 01:42:39.031749 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.032847 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.034003 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.034873 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.035500 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.036015 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.036589 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.037338 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.038830 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.039450 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.040444 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.041491 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.041831 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.043120 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.043786 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.044389 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.044901 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.046241 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.046896 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.048007 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.048388 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.050057 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 2 times. W0722 01:42:39.050415 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.051484 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.052219 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.052970 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.053709 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.054517 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.055269 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.056100 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.056443 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.059671 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.060459 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.060755 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.061589 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.062927 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.063869 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.064702 6706 consensus_peers.cc:390] Couldn't send request to peer peer-1 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. W0722 01:42:39.065254 6705 consensus_peers.cc:390] Couldn't send request to peer peer-0 for tablet TestTablet: Status: IO error: Artificial error caused by communication failure injection.. Retrying in the next heartbeat period. Already tried 1 times. I0722 01:42:39.094125 6627 raft_consensus.cc:510] T TestTablet P peer-0 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:39.094226 6627 raft_consensus_state.cc:226] T TestTablet P peer-0 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:39.094272 6627 raft_consensus_state.cc:244] T TestTablet P peer-0 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:39.094311 6627 raft_consensus_state.cc:248] T TestTablet P peer-0 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:39.094352 6627 raft_consensus.cc:521] T TestTablet P peer-0 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:39.094558 6627 raft_consensus.cc:510] T TestTablet P peer-1 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:39.094609 6627 raft_consensus_state.cc:226] T TestTablet P peer-1 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:39.094678 6627 raft_consensus_state.cc:244] T TestTablet P peer-1 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:39.094714 6627 raft_consensus_state.cc:248] T TestTablet P peer-1 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:39.094751 6627 raft_consensus.cc:521] T TestTablet P peer-1 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:39.094882 6627 raft_consensus.cc:510] T TestTablet P peer-2 [LEADER]: Raft consensus shutting down. I0722 01:42:39.095137 6627 consensus_peers.cc:411] Closing peer: peer-0 Waiting for outstanding requests to complete 0 I0722 01:42:39.095203 6627 consensus_peers.cc:411] Closing peer: peer-2 Waiting for outstanding requests to complete 0 I0722 01:42:39.095396 6627 consensus_peers.cc:411] Closing peer: peer-1 Waiting for outstanding requests to complete 0 I0722 01:42:39.095453 6627 raft_consensus_state.cc:226] T TestTablet P peer-2 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:39.095502 6627 raft_consensus_state.cc:244] T TestTablet P peer-2 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:39.095547 6627 raft_consensus_state.cc:248] T TestTablet P peer-2 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:39.095938 6627 raft_consensus.cc:521] T TestTablet P peer-2 [NON_PARTICIPANT]: Raft consensus Shutdown! [ OK ] RaftConsensusTest.TestReplicasHandleCommunicationErrors (135 ms) [ RUN ] RaftConsensusTest.TestLeaderHeartbeats I0722 01:42:39.114707 6627 raft_consensus.cc:66] T TestTablet P peer-0 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-0, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:39.114879 6627 raft_consensus.cc:66] T TestTablet P peer-1 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-1, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:39.115039 6627 raft_consensus.cc:66] T TestTablet P peer-2 [NON_PARTICIPANT]: Created Raft consensus for peer Replica: peer-2, State: 1, Role: NON_PARTICIPANT Watermarks: {Received: term: 0 index: 0 Replicated: term: 0 index: 0 Committed: term: 0 index: 0} Num. outstanding commits: 0 IsLocked: 0 I0722 01:42:39.115272 6627 raft_consensus.cc:80] T TestTablet P peer-0 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:39.115448 6627 raft_consensus.cc:80] T TestTablet P peer-1 [FOLLOWER]: Started. Quorum: seqno: 0 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:39.117126 6627 raft_consensus.cc:168] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): replicating to peers... I0722 01:42:39.118387 6627 raft_consensus.cc:180] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): committing... I0722 01:42:39.119313 6627 raft_consensus.cc:203] T TestTablet P peer-2 [LEADER]: ChangeConfiguration(op=term: 0 index: 1, seqno=1): a majority of peers have accepted the new configuration. Proceeding. I0722 01:42:39.119387 6627 raft_consensus.cc:80] T TestTablet P peer-2 [LEADER]: Started. Quorum: seqno: 1 peers { permanent_uuid: "peer-0" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-1" role: FOLLOWER last_known_addr { host: "0" port: 0 } } peers { permanent_uuid: "peer-2" role: LEADER last_known_addr { host: "0" port: 0 } } I0722 01:42:41.119838 6627 raft_consensus.cc:510] T TestTablet P peer-0 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:41.119932 6627 raft_consensus_state.cc:226] T TestTablet P peer-0 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:41.120009 6627 raft_consensus_state.cc:244] T TestTablet P peer-0 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:41.120048 6627 raft_consensus_state.cc:248] T TestTablet P peer-0 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:41.120090 6627 raft_consensus.cc:521] T TestTablet P peer-0 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:41.120276 6627 raft_consensus.cc:510] T TestTablet P peer-1 [FOLLOWER]: Raft consensus shutting down. I0722 01:42:41.120360 6627 raft_consensus_state.cc:226] T TestTablet P peer-1 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:41.120399 6627 raft_consensus_state.cc:244] T TestTablet P peer-1 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:41.120435 6627 raft_consensus_state.cc:248] T TestTablet P peer-1 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:41.120472 6627 raft_consensus.cc:521] T TestTablet P peer-1 [NON_PARTICIPANT]: Raft consensus Shutdown! I0722 01:42:41.120566 6627 raft_consensus.cc:510] T TestTablet P peer-2 [LEADER]: Raft consensus shutting down. I0722 01:42:41.120874 6627 consensus_peers.cc:411] Closing peer: peer-0 Waiting for outstanding requests to complete 0 I0722 01:42:41.120934 6627 consensus_peers.cc:411] Closing peer: peer-2 Waiting for outstanding requests to complete 0 I0722 01:42:41.121127 6627 consensus_peers.cc:411] Closing peer: peer-1 Waiting for outstanding requests to complete 0 I0722 01:42:41.121178 6627 raft_consensus_state.cc:226] T TestTablet P peer-2 [NON_PARTICIPANT]: Aborting pending transactions. I0722 01:42:41.121230 6627 raft_consensus_state.cc:244] T TestTablet P peer-2 [NON_PARTICIPANT]: Waiting on 0 outstanding commits. I0722 01:42:41.121276 6627 raft_consensus_state.cc:248] T TestTablet P peer-2 [NON_PARTICIPANT]: All local commits completed. I0722 01:42:41.121492 6627 raft_consensus.cc:521] T TestTablet P peer-2 [NON_PARTICIPANT]: Raft consensus Shutdown! [ OK ] RaftConsensusTest.TestLeaderHeartbeats (2025 ms) [----------] 6 tests from RaftConsensusTest (3287 ms total) [----------] Global test environment tear-down [==========] 6 tests from 1 test case ran. (3288 ms total) [ PASSED ] 6 tests. ThreadSanitizer: reported 8 warnings