Critical Description
Observed in ClientStressTest_LowMemory.TestMemoryThrottling.
=================================================================
==30312==ERROR: AddressSanitizer: heap-use-after-free on address 0x6020003f0f50 at pc 0x7f38bec95ddc bp 0x7f3873225950 sp 0x7f3873225948
READ of size 8 at 0x6020003f0f50 thread T139 (apply [worker]-)
#0 0x7f38bec95ddb in kudu::LongAdder::IncrementBy(long) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/striped64.cc:144:48
#1 0x7f38beb05abd in kudu::(anonymous namespace)::LRUCache::FreeEntry(kudu::(anonymous namespace)::LRUHandle*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/cache.cc:252:26
#2 0x7f38c83f0371 in kudu::cfile::BlockCacheHandle::Release() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/block_cache.h:178:28
#3 0x7f38c83f0249 in kudu::cfile::BlockCacheHandle::~BlockCacheHandle() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/block_cache.h:173:7
#4 0x7f38c83efec1 in std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex>::operator()(kudu::cfile::IndexTreeIterator::SeekedIndex*) const /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:67:2
#5 0x7f38c83efdea in std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex, std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> >::~unique_ptr() /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:184:4
#6 0x7f38c83efd14 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex, std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> >*>(std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex, std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> >*, std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex, std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> >*) /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_construct.h:103:6
#7 0x7f38c83efb6e in std::vector<std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex, std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> >, std::allocator<std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex, std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> > > >::~vector() /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_vector.h:415:9
#8 0x7f38c19ad25a in boost::optional_detail::optional_base<kudu::cfile::(anonymous namespace)::BloomCacheItem>::destroy_impl() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/optional/optional.hpp:566:49
#9 0x7f38c19ad458 in std::array<std::pair<unsigned long, boost::optional<kudu::cfile::(anonymous namespace)::BloomCacheItem> >, 4ul>::~array() /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/array:81:12
#10 0x7f38c19acff1 in void kudu::threadlocal::internal::Destroy<kudu::ThreadLocalCache<unsigned long, kudu::cfile::(anonymous namespace)::BloomCacheItem> >(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadlocal.h:136:3
#11 0x7f38becc145c in kudu::threadlocal::internal::InvokeDestructors(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadlocal.cc:41:5
#12 0x7f38c2373f81 in __nptl_deallocate_tsd /build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:158
#13 0x7f38c2374196 in start_thread /build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:325
#14 0x7f38bbd9737c in clone /build/eglibc-oGUzwX/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111
0x6020003f0f50 is located 0 bytes inside of 8-byte region [0x6020003f0f50,0x6020003f0f58)
freed by thread T139 (apply [worker]-) here:
#0 0x50f440 in operator delete(void*) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:110
#1 0x7f38becc145c in kudu::threadlocal::internal::InvokeDestructors(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadlocal.cc:41:5
#2 0x7f38c2373f81 in __nptl_deallocate_tsd /build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:158
previously allocated by thread T139 (apply [worker]-) here:
#0 0x50ed80 in operator new(unsigned long) /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:78
#1 0x7f38bec95ce1 in kudu::LongAdder::IncrementBy(long) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/striped64.cc:139:3
#2 0x7f38beb06ad7 in kudu::(anonymous namespace)::LRUCache::Lookup(kudu::Slice const&, unsigned int, bool) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/cache.cc:286:24
#3 0x7f38c19a4e8d in kudu::cfile::BlockCache::Lookup(kudu::cfile::BlockCache::CacheKey const&, kudu::Cache::CacheBehavior, kudu::cfile::BlockCacheHandle*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/block_cache.cc:78:30
#4 0x7f38c19bb7e4 in kudu::cfile::CFileReader::ReadBlock(kudu::cfile::BlockPointer const&, kudu::cfile::CFileReader::CacheControl, kudu::cfile::BlockHandle*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/cfile_reader.cc:340:14
#5 0x7f38c19f0d8f in kudu::cfile::IndexTreeIterator::LoadBlock(kudu::cfile::BlockPointer const&, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/index_btree.cc:270:3
#6 0x7f38c19f003d in kudu::cfile::IndexTreeIterator::SeekDownward(kudu::Slice const&, kudu::cfile::BlockPointer const&, int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/index_btree.cc:283:3
#7 0x7f38c19eff4f in kudu::cfile::IndexTreeIterator::SeekAtOrBefore(kudu::Slice const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/index_btree.cc:171:10
#8 0x7f38c19ac3eb in kudu::cfile::BloomFileReader::CheckKeyPresent(kudu::BloomKeyProbe const&, bool*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/bloomfile.cc:283:26
#9 0x7f38c82c78a2 in kudu::tablet::CFileSet::FindRow(kudu::tablet::RowSetKeyProbe const&, unsigned int*, kudu::tablet::ProbeStats*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/cfile_set.cc:216:31
#10 0x7f38c82c80b9 in kudu::tablet::CFileSet::CheckRowPresent(kudu::tablet::RowSetKeyProbe const&, bool*, unsigned int*, kudu::tablet::ProbeStats*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/cfile_set.cc:246:14
#11 0x7f38c832534a in kudu::tablet::DiskRowSet::CheckRowPresent(kudu::tablet::RowSetKeyProbe const&, bool*, kudu::tablet::ProbeStats*) const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/diskrowset.cc:646:3
#12 0x7f38c81e5911 in kudu::tablet::Tablet::BulkCheckPresence(kudu::tablet::WriteTransactionState*)::$_3::operator()() const /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/tablet.cc:767:7
#13 0x7f38c81e49b2 in kudu::tablet::Tablet::BulkCheckPresence(kudu::tablet::WriteTransactionState*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/tablet.cc:785:3
#14 0x7f38c81e6005 in kudu::tablet::Tablet::ApplyRowOperations(kudu::tablet::WriteTransactionState*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/tablet.cc:805:3
#15 0x7f38c82bcfd1 in kudu::tablet::WriteTransaction::Apply(gscoped_ptr<kudu::consensus::CommitMsg, kudu::DefaultDeleter<kudu::consensus::CommitMsg> >*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/transactions/write_transaction.cc:135:11
#16 0x7f38c82ab9df in kudu::tablet::TransactionDriver::ApplyTask() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/transactions/transaction_driver.cc:477:5
#17 0x7f38c82b0578 in kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::tablet::TransactionDriver::*)()>, void (kudu::tablet::TransactionDriver*)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::tablet::TransactionDriver::*)()>, kudu::tablet::TransactionDriver*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:871:14
#18 0x7f38c82b03d8 in kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::tablet::TransactionDriver::*)()>, void (kudu::tablet::TransactionDriver*), void (kudu::internal::UnretainedWrapper<kudu::tablet::TransactionDriver>)>, void (kudu::tablet::TransactionDriver*)>::Run(kudu::internal::BindStateBase*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:1063:12
#19 0x7f38c76695d7 in boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > > >::operator()() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
#20 0x7f38c5e1741e in boost::function0<void>::operator()() const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
#21 0x7f38becc4d5d in kudu::ThreadPool::DispatchThread(bool) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:347:23
#22 0x7f38beccbd67 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >::operator()() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
#23 0x7f38c5e1741e in boost::function0<void>::operator()() const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
#24 0x7f38becb1a83 in kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:590:3
#25 0x7f38c2374183 in start_thread /build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:312
Thread T139 (apply [worker]-) created by T75 (rpc worker-3039) here:
#0 0x4329e1 in __interceptor_pthread_create /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:245
#1 0x7f38becb0fbd in kudu::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:513:15
#2 0x7f38becc6c74 in kudu::Status kudu::Thread::Create<void (kudu::ThreadPool::*)(bool), kudu::ThreadPool*, bool>(std::string const&, std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool* const&, bool const&, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:164:12
#3 0x7f38becc2fe4 in kudu::ThreadPool::CreateThreadUnlocked() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:392:14
#4 0x7f38becc3bfd in kudu::ThreadPool::Submit(std::shared_ptr<kudu::Runnable>) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:218:21
#5 0x7f38becc38b3 in kudu::ThreadPool::SubmitFunc(boost::function<void ()>) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:186:10
#6 0x7f38becc36bf in kudu::ThreadPool::SubmitClosure(kudu::Callback<void ()> const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:182:10
#7 0x7f38c82aaf63 in kudu::tablet::TransactionDriver::ApplyAsync() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/transactions/transaction_driver.cc:458:23
#8 0x7f38c82a65dd in kudu::tablet::TransactionDriver::ReplicationFinished(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/transactions/transaction_driver.cc:412:5
#9 0x7f38c829d84c in kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (kudu::tablet::TransactionDriver::*)(kudu::Status const&)>, void (kudu::tablet::TransactionDriver*, kudu::Status const&)>::MakeItSo(kudu::internal::RunnableAdapter<void (kudu::tablet::TransactionDriver::*)(kudu::Status const&)>, kudu::tablet::TransactionDriver*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:887:14
#10 0x7f38c829d5a5 in kudu::internal::Invoker<1, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (kudu::tablet::TransactionDriver::*)(kudu::Status const&)>, void (kudu::tablet::TransactionDriver*, kudu::Status const&), void (kudu::internal::UnretainedWrapper<kudu::tablet::TransactionDriver>)>, void (kudu::tablet::TransactionDriver*, kudu::Status const&)>::Run(kudu::internal::BindStateBase*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:1116:12
#11 0x7f38c76c2be9 in kudu::consensus::PendingRounds::AdvanceCommittedIndex(long) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/consensus/raft_consensus_state.cc:508:12
#12 0x7f38c76968c5 in kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/consensus/raft_consensus.cc:1295:5
#13 0x7f38c7694a2c in kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/consensus/raft_consensus.cc:752:14
#14 0x7f38ca53651b in kudu::tserver::ConsensusServiceImpl::UpdateConsensus(kudu::consensus::ConsensusRequestPB const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tserver/tablet_service.cc:843:25
#15 0x7f38c6a2a18a in std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), kudu::consensus::ConsensusServiceIf::ConsensusServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/functional:2071:2
#16 0x7f38c5e90a94 in std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/functional:2471:14
#17 0x7f38c5e8fac6 in kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/service_if.cc:134:3
#18 0x7f38c5e92de8 in kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/service_pool.cc:210:15
#19 0x7f38c5e96f67 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ServicePool>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ServicePool*> > >::operator()() /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
#20 0x7f38c5e1741e in boost::function0<void>::operator()() const /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
#21 0x7f38becb1a83 in kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:590:3
#22 0x7f38c2374183 in start_thread /build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:312
Thread T75 (rpc worker-3039) created by T0 here:
#0 0x4329e1 in __interceptor_pthread_create /home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:245
#1 0x7f38becb0fbd in kudu::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:513:15
#2 0x7f38c5e95284 in kudu::Status kudu::Thread::Create<void (kudu::rpc::ServicePool::*)(), kudu::rpc::ServicePool*>(std::string const&, std::string const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:158:12
#3 0x7f38c5e92814 in kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/service_pool.cc:81:5
#4 0x7f38c915aecb in kudu::RpcServer::RegisterService(gscoped_ptr<kudu::rpc::ServiceIf, kudu::DefaultDeleter<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/server/rpc_server.cc:122:3
#5 0x7f38c91660e3 in kudu::server::ServerBase::RegisterService(gscoped_ptr<kudu::rpc::ServiceIf, kudu::DefaultDeleter<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/server/server_base.cc:364:23
#6 0x7f38ca52c173 in kudu::tserver::TabletServer::Start() /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tserver/tablet_server.cc:117:3
#7 0x5125fa in kudu::tserver::TabletServerMain(int, char**) /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tserver/tablet_server_main.cc:77:3
#8 0x5120f4 in main /home/jenkins-slave/workspace/kudu-master/3/src/kudu/tserver/tablet_server_main.cc:91:10
#9 0x7f38bbcbef44 in __libc_start_main /build/eglibc-oGUzwX/eglibc-2.19/csu/libc-start.c:287
SUMMARY: AddressSanitizer: heap-use-after-free /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/striped64.cc:144:48 in kudu::LongAdder::IncrementBy(long)
Shadow bytes around the buggy address:
0x0c0480076190: fa fa fd fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04800761a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04800761b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04800761c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04800761d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c04800761e0: fa fa fa fa fa fa fa fa fa fa[fd]fa fa fa 00 fa
0x0c04800761f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480076200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480076210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480076220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480076230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==30312==ABORTING
Attachments
Issue Links
- duplicates
-
KUDU-1992 heap-use-after-free when running threadlocal cache destructor
-
- Resolved
-