[==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from UpdateScanDeltaCompactionTest [ RUN ] UpdateScanDeltaCompactionTest.TestAll I1216 18:23:11.573256 30436 mem_tracker.cc:118] MemTracker: hard memory limit is 23.413431 GB I1216 18:23:11.573729 30436 mem_tracker.cc:120] MemTracker: soft memory limit is 14.048059 GB I1216 18:23:11.591179 30436 hybrid_clock.cc:176] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 174711 I1216 18:23:11.599450 30436 server_base.cc:162] Could not load existing FS layout: Not found: /data2/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1450318991557639-30436/minicluster-data/master-0-root/instance: No such file or directory (error 2) I1216 18:23:11.599761 30436 server_base.cc:163] Creating new FS layout I1216 18:23:11.611913 30436 fs_manager.cc:343] Generated new instance metadata in path /data2/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1450318991557639-30436/minicluster-data/master-0-root/instance: uuid: "e87276f8462046038be791e3f9e51bd4" format_stamp: "Formatted at 2015-12-17 02:23:11 on kudu-cpp11.vpc.cloudera.com" I1216 18:23:11.622906 30436 fs_manager.cc:240] Opened local filesystem: /data2/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1450318991557639-30436/minicluster-data/master-0-root uuid: "e87276f8462046038be791e3f9e51bd4" format_stamp: "Formatted at 2015-12-17 02:23:11 on kudu-cpp11.vpc.cloudera.com" I1216 18:23:11.729110 30436 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:46884 I1216 18:23:11.731359 30436 webserver.cc:133] Starting webserver on 0.0.0.0:0 I1216 18:23:11.731539 30436 webserver.cc:144] Document root disabled I1216 18:23:11.734211 30436 webserver.cc:227] Webserver started. Bound to: http://0.0.0.0:34852/ I1216 18:23:11.763960 30481 tablet_bootstrap.cc:358] T 00000000000000000000000000000000 P e87276f8462046038be791e3f9e51bd4: Bootstrap starting. I1216 18:23:11.769592 30481 tablet_bootstrap.cc:521] T 00000000000000000000000000000000 P e87276f8462046038be791e3f9e51bd4: Time spent opening tablet: real 0.002s user 0.002s sys 0.000s I1216 18:23:11.770318 30481 tablet_bootstrap.cc:461] T 00000000000000000000000000000000 P e87276f8462046038be791e3f9e51bd4: No blocks or log segments found. Creating new log. I1216 18:23:11.772591 30481 log.cc:333] Log is configured to *not* fsync() on all Append() calls I1216 18:23:11.818966 30481 tablet_bootstrap.cc:358] T 00000000000000000000000000000000 P e87276f8462046038be791e3f9e51bd4: No bootstrap required, opened a new log I1216 18:23:11.823675 30481 local_consensus.cc:58] T 00000000000000000000000000000000 P e87276f8462046038be791e3f9e51bd4: Starting LocalConsensus... I1216 18:23:11.824157 30481 sys_catalog.cc:225] T 00000000000000000000000000000000 P e87276f8462046038be791e3f9e51bd4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 0 leader_uuid: "e87276f8462046038be791e3f9e51bd4" config { opid_index: -1 local: true peers { permanent_uuid: "e87276f8462046038be791e3f9e51bd4" member_type: VOTER } } I1216 18:23:11.824776 30481 sys_catalog.cc:228] T 00000000000000000000000000000000 P e87276f8462046038be791e3f9e51bd4 [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER I1216 18:23:11.829080 30481 sys_catalog.cc:297] T 00000000000000000000000000000000 P e87276f8462046038be791e3f9e51bd4 [sys.catalog]: configured and running, proceeding with master startup. I1216 18:23:11.829289 30484 catalog_manager.cc:514] Loading table and tablet metadata into memory... I1216 18:23:11.854202 30436 hybrid_clock.cc:176] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 174711 I1216 18:23:11.859084 30436 server_base.cc:162] Could not load existing FS layout: Not found: /data2/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1450318991557639-30436/minicluster-data/ts-0-root/instance: No such file or directory (error 2) I1216 18:23:11.859254 30436 server_base.cc:163] Creating new FS layout I1216 18:23:11.864740 30436 fs_manager.cc:343] Generated new instance metadata in path /data2/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1450318991557639-30436/minicluster-data/ts-0-root/instance: uuid: "679694f8ad274496abdcad538b12fcf1" format_stamp: "Formatted at 2015-12-17 02:23:11 on kudu-cpp11.vpc.cloudera.com" I1216 18:23:11.872964 30436 fs_manager.cc:240] Opened local filesystem: /data2/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1450318991557639-30436/minicluster-data/ts-0-root uuid: "679694f8ad274496abdcad538b12fcf1" format_stamp: "Formatted at 2015-12-17 02:23:11 on kudu-cpp11.vpc.cloudera.com" I1216 18:23:11.980624 30436 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:36395 I1216 18:23:11.982319 30436 webserver.cc:133] Starting webserver on 0.0.0.0:0 I1216 18:23:11.982448 30436 webserver.cc:144] Document root disabled I1216 18:23:11.984494 30436 webserver.cc:227] Webserver started. Bound to: http://0.0.0.0:58851/ W1216 18:23:13.508325 30445 thread.cc:431] negotiator [worker] (thread pool) Time spent starting thread: real 1.516s user 0.002s sys 0.001s I1216 18:23:13.524235 30547 heartbeater.cc:256] Connected to a leader master server at 127.0.0.1:46884 I1216 18:23:13.524873 30547 heartbeater.cc:326] Registering TS with master... I1216 18:23:13.525732 30547 heartbeater.cc:332] Sending a full tablet report to master... I1216 18:23:13.528555 30450 ts_manager.cc:80] Registered new tablet server { permanent_uuid: "679694f8ad274496abdcad538b12fcf1" instance_seqno: 1450318991961037 } with Master I1216 18:23:13.533711 30436 mini_cluster.cc:291] 1 TS(s) registered with Master after 1.54563s I1216 18:23:13.556035 30452 catalog_manager.cc:672] CreateTable from {real_user=dan, eff_user=} at 127.0.0.1:43621: name: "update-scan-delta-compact-tbl" schema { columns { name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int64" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } split_rows { } partition_schema { } I1216 18:23:13.617691 30505 tablet_service.cc:598] Processing CreateTablet for tablet 7b5b4651c246417b8a22a497c0eb9fdb (table=update-scan-delta-compact-tbl [id=bf5521d2a5e84f669b20874aea143d88]), partition=range: [(), ()) I1216 18:23:13.626315 30505 ts_tablet_manager.cc:730] Registered tablet 7b5b4651c246417b8a22a497c0eb9fdb I1216 18:23:13.628882 30572 ts_tablet_manager.cc:602] T 7b5b4651c246417b8a22a497c0eb9fdb P 679694f8ad274496abdcad538b12fcf1: Bootstrapping tablet I1216 18:23:13.637255 30572 tablet_bootstrap.cc:358] T 7b5b4651c246417b8a22a497c0eb9fdb P 679694f8ad274496abdcad538b12fcf1: Bootstrap starting. I1216 18:23:13.642696 30572 tablet_bootstrap.cc:521] T 7b5b4651c246417b8a22a497c0eb9fdb P 679694f8ad274496abdcad538b12fcf1: Time spent opening tablet: real 0.001s user 0.000s sys 0.001s I1216 18:23:13.643303 30572 tablet_bootstrap.cc:461] T 7b5b4651c246417b8a22a497c0eb9fdb P 679694f8ad274496abdcad538b12fcf1: No blocks or log segments found. Creating new log. I1216 18:23:13.653506 30572 tablet_bootstrap.cc:358] T 7b5b4651c246417b8a22a497c0eb9fdb P 679694f8ad274496abdcad538b12fcf1: No bootstrap required, opened a new log I1216 18:23:13.655328 30572 ts_tablet_manager.cc:607] T 7b5b4651c246417b8a22a497c0eb9fdb P 679694f8ad274496abdcad538b12fcf1: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.012s I1216 18:23:13.658843 30572 local_consensus.cc:58] T 7b5b4651c246417b8a22a497c0eb9fdb P 679694f8ad274496abdcad538b12fcf1: Starting LocalConsensus... I1216 18:23:13.661614 30572 ts_tablet_manager.cc:629] T 7b5b4651c246417b8a22a497c0eb9fdb P 679694f8ad274496abdcad538b12fcf1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s I1216 18:23:13.663569 30452 catalog_manager.cc:1546] Tablet: 7b5b4651c246417b8a22a497c0eb9fdb reported consensus state change. New consensus state: current_term: 0 leader_uuid: "679694f8ad274496abdcad538b12fcf1" config { opid_index: -1 local: true peers { permanent_uuid: "679694f8ad274496abdcad538b12fcf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36395 } } } W1216 18:23:14.164870 30566 outbound_call.cc:185] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.0.1:36395, user_credentials={real_user=dan, eff_user=}} blocked reactor thread for 457957us W1216 18:23:14.230417 30564 kernel_stack_watchdog.cc:122] Thread 30566 stuck at src/kudu/rpc/outbound_call.cc:171 for 283ms: Kernel stack: [] pipe_wait+0x5b/0x80 [] pipe_read+0x367/0x4f0 [] do_sync_read+0xfa/0x140 [] vfs_read+0xb5/0x1a0 [] sys_read+0x51/0x90 [] system_call_fastpath+0x16/0x1b [] 0xffffffffffffffff User stack: @ 0x7f443946a7e1 __tsan::ProcessPendingSignals() at ??:0 @ 0x7f44394b54c4 __tsan_atomic32_compare_exchange_val at ??:0 @ 0x7f4432f071a5 std::tr1::_Sp_counted_base<>::_M_add_ref_lock() at ??:0 @ 0x7f4432f07107 std::tr1::__shared_count<>::__shared_count() at ??:0 @ 0x7f4432f0702e std::tr1::__shared_ptr<>::__shared_ptr<>() at ??:0 @ 0x7f4432f06edc std::tr1::shared_ptr<>::shared_ptr<>() at ??:0 @ 0x7f4432ef7156 std::tr1::weak_ptr<>::lock() at ??:0 @ 0x7f4432ef02dd kudu::client::internal::Batcher::CheckForFinishedFlush() at ??:0 @ 0x7f4432eee754 kudu::client::internal::Batcher::ProcessWriteResponse() at ??:0 @ 0x7f4432eecd5c kudu::client::internal::WriteRpc::SendRpcCb() at ??:0 @ 0x7f4432f0296c boost::_mfi::mf1<>::operator()() at ??:0 @ 0x7f4432f02838 boost::_bi::list2<>::operator()<>() at ??:0 @ 0x7f4432f0274b boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f4432f021c7 boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f44308f059b boost::function0<>::operator()() at ??:0 @ 0x7f44308ecdd5 kudu::rpc::OutboundCall::CallCallback() at ??:0 I1216 18:23:14.877362 30436 update_scan_delta_compact-test.cc:181] Time spent Insert: real 1.197s user 0.717s sys 0.112s I1216 18:23:16.061780 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 1.179s user 0.006s sys 0.004s I1216 18:23:16.163977 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.100s user 0.003s sys 0.002s I1216 18:23:16.245214 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.081s user 0.003s sys 0.000s I1216 18:23:16.330404 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.085s user 0.000s sys 0.004s I1216 18:23:16.439182 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.108s user 0.002s sys 0.005s I1216 18:23:16.490592 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.051s user 0.004s sys 0.002s I1216 18:23:16.559593 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.068s user 0.003s sys 0.003s I1216 18:23:16.698521 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.138s user 0.006s sys 0.000s I1216 18:23:16.798674 30599 update_scan_delta_compact-test.cc:243] Time spent Update: real 1.918s user 0.543s sys 0.144s I1216 18:23:16.829560 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.130s user 0.002s sys 0.004s I1216 18:23:17.342067 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.512s user 0.005s sys 0.003s ================== WARNING: ThreadSanitizer: data race (pid=30436) Read of size 4 at 0x7d44000bde30 by thread T131 (mutexes: write M97279): #0 google::protobuf::internal::GeneratedMessageReflection::HasBit(google::protobuf::Message const&, google::protobuf::FieldDescriptor const*) const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/generated_message_reflection.cc:1531 (libprotobuf.so.9+0x0000001044ed) #1 google::protobuf::TextFormat::Printer::Print(google::protobuf::Message const&, google::protobuf::TextFormat::Printer::TextGenerator&) const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/text_format.cc:1422 (libprotobuf.so.9+0x00000011a852) #2 google::protobuf::TextFormat::Printer::Print(google::protobuf::Message const&, google::protobuf::io::ZeroCopyOutputStream*) const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/text_format.cc:1391 (libprotobuf.so.9+0x000000118274) #3 google::protobuf::Message::ShortDebugString() const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/text_format.cc:88 (libprotobuf.so.9+0x000000118076) #4 kudu::tablet::WriteTransactionState::ToString() const /data1/kudu/src/kudu/tablet/transactions/write_transaction.cc:366 (libtablet.so+0x0000002e34a9) #5 kudu::tablet::WriteTransaction::ToString() const /data1/kudu/src/kudu/tablet/transactions/write_transaction.cc:203 (libtablet.so+0x0000002e2285) #6 kudu::tablet::TransactionDriver::ToStringUnlocked() const /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:118 (libtablet.so+0x0000002cd039) #7 kudu::tablet::TransactionDriver::ToString() const /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:112 (libtablet.so+0x0000002ccf27) #8 kudu::tablet::TabletPeer::GetInFlightTransactions(kudu::tablet::Transaction::TraceType, std::vector >*) const /data1/kudu/src/kudu/tablet/tablet_peer.cc:410 (libtablet.so+0x0000002b9e0e) #9 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) /data1/kudu/src/kudu/tserver/tserver-path-handlers.cc:130 (libtserver.so+0x0000001f092e) #10 boost::_mfi::mf2, std::allocator >*>::operator()(kudu::tserver::TabletServerPathHandlers*, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) const /usr/include/boost/bind/mem_fn_template.hpp:274 (libtserver.so+0x0000001fa47b) #11 void boost::_bi::list3, boost::arg<1>, boost::arg<2> >::operator(), std::allocator >*>, boost::_bi::list2, std::allocator >*&> >(boost::_bi::type, boost::_mfi::mf2, std::allocator >*>&, boost::_bi::list2, std::allocator >*&>&, int) /usr/include/boost/bind/bind.hpp:385 (libtserver.so+0x0000001fa261) #12 void boost::_bi::bind_t, std::allocator >*>, boost::_bi::list3, boost::arg<1>, boost::arg<2> > >::operator(), std::allocator >*>(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*&) /usr/include/boost/bind/bind_template.hpp:76 (libtserver.so+0x0000001fa11a) #13 boost::detail::function::void_function_obj_invoker2, std::allocator >*>, boost::_bi::list3, boost::arg<1>, boost::arg<2> > >, void, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*>::invoke(boost::detail::function::function_buffer&, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) /usr/include/boost/function/function_template.hpp:153 (libtserver.so+0x0000001f9dac) #14 boost::function2, std::allocator >*>::operator()(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) const /usr/include/boost/function/function_template.hpp:1012 (libserver_process.so+0x0000000bb78a) #15 kudu::Webserver::RunPathHandler(kudu::Webserver::PathHandler const&, sq_connection*, sq_request_info*) /data1/kudu/src/kudu/server/webserver.cc:354 (libserver_process.so+0x0000000b9027) #16 kudu::Webserver::BeginRequestCallback(sq_connection*, sq_request_info*) /data1/kudu/src/kudu/server/webserver.cc:298 (libserver_process.so+0x0000000b8637) #17 kudu::Webserver::BeginRequestCallbackStatic(sq_connection*) /data1/kudu/src/kudu/server/webserver.cc:273 (libserver_process.so+0x0000000b7dbd) #18 handle_request squeasel.c (libserver_process.so+0x0000000d64ff) Previous write of size 1 at 0x7d44000bde32 by thread T121 (mutexes: write M97294): #0 memset /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:586 (update_scan_delta_compact-test+0x00000005c43b) #1 kudu::consensus::OpId::Clear() /data1/kudu/src/kudu/consensus/opid.pb.cc:178 (libconsensus_metadata_proto.so+0x00000001dcc4) #2 kudu::consensus::OpId::CopyFrom(kudu::consensus::OpId const&) /data1/kudu/src/kudu/consensus/opid.pb.cc:347 (libconsensus_metadata_proto.so+0x00000001ee53) #3 kudu::tablet::TransactionDriver::ReplicationFinished(kudu::Status const&) /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:272 (libtablet.so+0x0000002cc88f) #4 kudu::internal::RunnableAdapter::Run(kudu::tablet::TransactionDriver*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:190 (libtablet.so+0x0000002c569b) #5 kudu::internal::InvokeHelper, void (kudu::tablet::TransactionDriver*, kudu::Status const&)>::MakeItSo(kudu::internal::RunnableAdapter, kudu::tablet::TransactionDriver*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:887 (libtablet.so+0x0000002c54db) #6 kudu::internal::Invoker<1, kudu::internal::BindState, void (kudu::tablet::TransactionDriver*, kudu::Status const&), void (kudu::internal::UnretainedWrapper)>, void (kudu::tablet::TransactionDriver*, kudu::Status const&)>::Run(kudu::internal::BindStateBase*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:1116 (libtablet.so+0x0000002c53f5) #7 kudu::Callback::Run(kudu::Status const&) const /data1/kudu/src/kudu/gutil/callback.h:436 (libconsensus.so+0x0000000aa928) #8 kudu::consensus::ConsensusRound::NotifyReplicationFinished(kudu::Status const&) /data1/kudu/src/kudu/consensus/consensus.cc:59 (libconsensus.so+0x0000000a96f8) #9 kudu::internal::RunnableAdapter::Run(kudu::consensus::ConsensusRound*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:190 (libconsensus.so+0x00000010c8db) #10 kudu::internal::InvokeHelper, void (kudu::consensus::ConsensusRound*, kudu::Status const&)>::MakeItSo(kudu::internal::RunnableAdapter, kudu::consensus::ConsensusRound*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:887 (libconsensus.so+0x00000010c75b) #11 kudu::internal::Invoker<1, kudu::internal::BindState, void (kudu::consensus::ConsensusRound*, kudu::Status const&), void (scoped_refptr)>, void (kudu::consensus::ConsensusRound*, kudu::Status const&)>::Run(kudu::internal::BindStateBase*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:1116 (libconsensus.so+0x00000010c677) #12 kudu::Callback::Run(kudu::Status const&) const /data1/kudu/src/kudu/gutil/callback.h:436 (libconsensus.so+0x0000000aa928) #13 kudu::log::Log::AppendThread::RunThread() /data1/kudu/src/kudu/consensus/log.cc:218 (liblog.so+0x00000007fa51) #14 boost::_mfi::mf0::operator()(kudu::log::Log::AppendThread*) const /usr/include/boost/bind/mem_fn_template.hpp:49 (liblog.so+0x0000000a257b) #15 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:246 (liblog.so+0x0000000a2458) #16 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (liblog.so+0x0000000a238a) #17 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (liblog.so+0x0000000a1f80) #18 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #19 kudu::Thread::SuperviseThread(void*) /data1/kudu/src/kudu/util/thread.cc:507 (libkudu_util.so+0x0000002d8819) Location is heap block of size 288 at 0x7d44000bdd80 allocated by thread T60: #0 operator new(unsigned long) /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:37 (update_scan_delta_compact-test+0x0000000b8453) #1 kudu::tserver::TabletServiceImpl::Write(kudu::tserver::WriteRequestPB const*, kudu::tserver::WriteResponsePB*, kudu::rpc::RpcContext*) /data1/kudu/src/kudu/tserver/tablet_service.cc:711 (libtserver.so+0x000000192a83) #2 kudu::tserver::TabletServerServiceIf::Handle(kudu::rpc::InboundCall*) /data1/kudu/src/kudu/tserver/tserver_service.service.cc:83 (libtserver_service_proto.so+0x00000001b1f8) #3 kudu::rpc::ServicePool::RunThread() /data1/kudu/src/kudu/rpc/service_pool.cc:172 (libkrpc.so+0x0000001365a3) #4 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /usr/include/boost/bind/mem_fn_template.hpp:49 (libkrpc.so+0x000000139e3b) #5 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:246 (libkrpc.so+0x000000139d18) #6 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkrpc.so+0x000000139c7a) #7 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkrpc.so+0x000000139930) #8 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #9 kudu::Thread::SuperviseThread(void*) /data1/kudu/src/kudu/util/thread.cc:507 (libkudu_util.so+0x0000002d8819) Mutex M97279 (0x7d280006df00) created at: #0 __tsan_atomic32_compare_exchange_strong /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:779 (update_scan_delta_compact-test+0x00000007d488) #1 base::subtle::Acquire_CompareAndSwap(int volatile*, int, int) /data1/kudu/src/kudu/gutil/atomicops-internals-tsan.h:85 (libmaster.so+0x000000116375) #2 base::SpinLock::Lock() /data1/kudu/src/kudu/gutil/spinlock.h:73 (libmaster.so+0x00000015c033) #3 kudu::simple_spinlock::lock() /data1/kudu/src/kudu/util/locks.h:42 (libmaster.so+0x00000015bfe8) #4 lock_guard /usr/include/boost/thread/locks.hpp:194 (libmaster.so+0x000000116f55) #5 kudu::tablet::TransactionDriver::PrepareAndStart() /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:175 (libtablet.so+0x0000002cf511) #6 kudu::tablet::TransactionDriver::PrepareAndStartTask() /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:153 (libtablet.so+0x0000002ce304) #7 kudu::internal::RunnableAdapter::Run(kudu::tablet::TransactionDriver*) /data1/kudu/src/kudu/gutil/bind_internal.h:134 (libtablet.so+0x0000002d47cb) #8 kudu::internal::InvokeHelper, void (kudu::tablet::TransactionDriver*)>::MakeItSo(kudu::internal::RunnableAdapter, kudu::tablet::TransactionDriver*) /data1/kudu/src/kudu/gutil/bind_internal.h:871 (libtablet.so+0x0000002d46fb) #9 kudu::internal::Invoker<1, kudu::internal::BindState, void (kudu::tablet::TransactionDriver*), void (kudu::internal::UnretainedWrapper)>, void (kudu::tablet::TransactionDriver*)>::Run(kudu::internal::BindStateBase*) /data1/kudu/src/kudu/gutil/bind_internal.h:1063 (libtablet.so+0x0000002d4682) #10 kudu::Callback::Run() const /data1/kudu/src/kudu/gutil/callback.h:396 (libconsensus.so+0x0000000f8efb) #11 boost::_mfi::cmf0 >::operator()(kudu::Callback const&) const /usr/include/boost/bind/mem_fn_template.hpp:117 (libconsensus.so+0x0000001046cb) #12 void boost::_bi::list1 > >::operator() >, boost::_bi::list0>(boost::_bi::type, boost::_mfi::cmf0 >&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:246 (libconsensus.so+0x0000001045a5) #13 boost::_bi::bind_t >, boost::_bi::list1 > > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libconsensus.so+0x00000010451a) #14 boost::detail::function::void_function_obj_invoker0 >, boost::_bi::list1 > > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libconsensus.so+0x000000104056) #15 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #16 kudu::FunctionRunnable::Run() /data1/kudu/src/kudu/util/threadpool.cc:46 (libkudu_util.so+0x0000002eb9dc) #17 kudu::ThreadPool::DispatchThread(bool) /data1/kudu/src/kudu/util/threadpool.cc:317 (libkudu_util.so+0x0000002e9b91) #18 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:162 (libkudu_util.so+0x0000002ef171) #19 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306 (libkudu_util.so+0x0000002ef03f) #20 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x0000002eef6a) #21 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x0000002eeae6) #22 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #23 kudu::Thread::SuperviseThread(void*) /data1/kudu/src/kudu/util/thread.cc:507 (libkudu_util.so+0x0000002d8819) Mutex M97294 (0x7d280006df30) created at: #0 __tsan_atomic32_compare_exchange_strong /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:779 (update_scan_delta_compact-test+0x00000007d488) #1 base::subtle::Acquire_CompareAndSwap(int volatile*, int, int) /data1/kudu/src/kudu/gutil/atomicops-internals-tsan.h:85 (libmaster.so+0x000000116375) #2 base::SpinLock::Lock() /data1/kudu/src/kudu/gutil/spinlock.h:73 (libmaster.so+0x00000015c033) #3 kudu::simple_spinlock::lock() /data1/kudu/src/kudu/util/locks.h:42 (libmaster.so+0x00000015bfe8) #4 lock_guard /usr/include/boost/thread/locks.hpp:194 (libmaster.so+0x000000116f55) #5 kudu::tablet::TransactionDriver::GetOpId() /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:94 (libtablet.so+0x0000002ccd8a) #6 kudu::tablet::TabletPeer::GetInFlightTransactions(kudu::tablet::Transaction::TraceType, std::vector >*) const /data1/kudu/src/kudu/tablet/tablet_peer.cc:401 (libtablet.so+0x0000002b9d06) #7 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) /data1/kudu/src/kudu/tserver/tserver-path-handlers.cc:130 (libtserver.so+0x0000001f092e) #8 boost::_mfi::mf2, std::allocator >*>::operator()(kudu::tserver::TabletServerPathHandlers*, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) const /usr/include/boost/bind/mem_fn_template.hpp:274 (libtserver.so+0x0000001fa47b) #9 void boost::_bi::list3, boost::arg<1>, boost::arg<2> >::operator(), std::allocator >*>, boost::_bi::list2, std::allocator >*&> >(boost::_bi::type, boost::_mfi::mf2, std::allocator >*>&, boost::_bi::list2, std::allocator >*&>&, int) /usr/include/boost/bind/bind.hpp:385 (libtserver.so+0x0000001fa261) #10 void boost::_bi::bind_t, std::allocator >*>, boost::_bi::list3, boost::arg<1>, boost::arg<2> > >::operator(), std::allocator >*>(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*&) /usr/include/boost/bind/bind_template.hpp:76 (libtserver.so+0x0000001fa11a) #11 boost::detail::function::void_function_obj_invoker2, std::allocator >*>, boost::_bi::list3, boost::arg<1>, boost::arg<2> > >, void, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*>::invoke(boost::detail::function::function_buffer&, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) /usr/include/boost/function/function_template.hpp:153 (libtserver.so+0x0000001f9dac) #12 boost::function2, std::allocator >*>::operator()(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) const /usr/include/boost/function/function_template.hpp:1012 (libserver_process.so+0x0000000bb78a) #13 kudu::Webserver::RunPathHandler(kudu::Webserver::PathHandler const&, sq_connection*, sq_request_info*) /data1/kudu/src/kudu/server/webserver.cc:354 (libserver_process.so+0x0000000b9027) #14 kudu::Webserver::BeginRequestCallback(sq_connection*, sq_request_info*) /data1/kudu/src/kudu/server/webserver.cc:298 (libserver_process.so+0x0000000b8637) #15 kudu::Webserver::BeginRequestCallbackStatic(sq_connection*) /data1/kudu/src/kudu/server/webserver.cc:273 (libserver_process.so+0x0000000b7dbd) #16 handle_request squeasel.c (libserver_process.so+0x0000000d64ff) Thread T131 'sq_worker' (tid=30604, running) created by thread T105 at: #0 pthread_create /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:849 (update_scan_delta_compact-test+0x000000059fd3) #1 master_thread squeasel.c (libserver_process.so+0x0000000d3b51) Thread T121 'appender-30575' (tid=30575, running) created by thread T118 at: #0 pthread_create /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:849 (update_scan_delta_compact-test+0x000000059fd3) #1 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /data1/kudu/src/kudu/util/thread.cc:435 (libkudu_util.so+0x0000002d7dbc) #2 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::log::Log::AppendThread::* const&)(), kudu::log::Log::AppendThread* const&, scoped_refptr*) /data1/kudu/src/kudu/util/thread.h:137 (liblog.so+0x0000000900c6) #3 kudu::log::Log::AppendThread::Init() /data1/kudu/src/kudu/consensus/log.cc:150 (liblog.so+0x00000007e175) #4 kudu::log::Log::Init() /data1/kudu/src/kudu/consensus/log.cc:341 (liblog.so+0x0000000846ed) #5 kudu::log::Log::Open(kudu::log::LogOptions const&, kudu::FsManager*, std::string const&, kudu::Schema const&, unsigned int, scoped_refptr const&, scoped_refptr*) /data1/kudu/src/kudu/consensus/log.cc:271 (liblog.so+0x0000000837d5) #6 kudu::tablet::TabletBootstrap::OpenNewLog() /data1/kudu/src/kudu/tablet/tablet_bootstrap.cc:639 (libtablet.so+0x000000289f91) #7 kudu::tablet::TabletBootstrap::Bootstrap(std::shared_ptr*, scoped_refptr*, kudu::consensus::ConsensusBootstrapInfo*) /data1/kudu/src/kudu/tablet/tablet_bootstrap.cc:462 (libtablet.so+0x0000002867e8) #8 kudu::tablet::BootstrapTablet(scoped_refptr const&, scoped_refptr const&, std::shared_ptr const&, kudu::MetricRegistry*, kudu::tablet::TabletStatusListener*, std::shared_ptr*, scoped_refptr*, scoped_refptr const&, kudu::consensus::ConsensusBootstrapInfo*) /data1/kudu/src/kudu/tablet/tablet_bootstrap.cc:377 (libtablet.so+0x0000002852fc) #9 kudu::tserver::TSTabletManager::OpenTablet(scoped_refptr const&, scoped_refptr const&) /data1/kudu/src/kudu/tserver/ts_tablet_manager.cc:611 (libtserver.so+0x0000001c9450) #10 boost::_mfi::mf2 const&, scoped_refptr const&>::operator()(kudu::tserver::TSTabletManager*, scoped_refptr const&, scoped_refptr const&) const /usr/include/boost/bind/mem_fn_template.hpp:274 (libtserver.so+0x0000001e0b8b) #11 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator() const&, scoped_refptr const&>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2 const&, scoped_refptr const&>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:385 (libtserver.so+0x0000001e0a53) #12 boost::_bi::bind_t const&, scoped_refptr const&>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libtserver.so+0x0000001e097a) #13 boost::detail::function::void_function_obj_invoker0 const&, scoped_refptr const&>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libtserver.so+0x0000001e04b6) #14 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #15 kudu::FunctionRunnable::Run() /data1/kudu/src/kudu/util/threadpool.cc:46 (libkudu_util.so+0x0000002eb9dc) #16 kudu::ThreadPool::DispatchThread(bool) /data1/kudu/src/kudu/util/threadpool.cc:317 (libkudu_util.so+0x0000002e9b91) #17 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:162 (libkudu_util.so+0x0000002ef171) #18 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306 (libkudu_util.so+0x0000002ef03f) #19 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x0000002eef6a) #20 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x0000002eeae6) #21 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #22 kudu::Thread::SuperviseThread(void*) /data1/kudu/src/kudu/util/thread.cc:507 (libkudu_util.so+0x0000002d8819) Thread T60 'rpc worker-3050' (tid=30501, running) created by main thread at: #0 pthread_create /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:849 (update_scan_delta_compact-test+0x000000059fd3) #1 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /data1/kudu/src/kudu/util/thread.cc:435 (libkudu_util.so+0x0000002d7dbc) #2 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /data1/kudu/src/kudu/util/thread.h:137 (libkrpc.so+0x000000137c26) #3 kudu::rpc::ServicePool::Init(int) /data1/kudu/src/kudu/rpc/service_pool.cc:76 (libkrpc.so+0x000000135ce4) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /data1/kudu/src/kudu/server/rpc_server.cc:122 (libserver_process.so+0x0000000898c8) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /data1/kudu/src/kudu/server/server_base.cc:237 (libserver_process.so+0x000000095c7e) #6 kudu::tserver::TabletServer::Start() /data1/kudu/src/kudu/tserver/tablet_server.cc:117 (libtserver.so+0x000000188726) #7 kudu::tserver::MiniTabletServer::Start() /data1/kudu/src/kudu/tserver/mini_tablet_server.cc:77 (libtserver.so+0x00000011b747) #8 kudu::MiniCluster::AddTabletServer() /data1/kudu/src/kudu/integration-tests/mini_cluster.cc:173 (libintegration-tests.so+0x0000000d5401) #9 kudu::MiniCluster::Start() /data1/kudu/src/kudu/integration-tests/mini_cluster.cc:89 (libintegration-tests.so+0x0000000d36bb) #10 kudu::tablet::UpdateScanDeltaCompactionTest::InitCluster() /data1/kudu/src/kudu/integration-tests/update_scan_delta_compact-test.cc:113 (update_scan_delta_compact-test+0x0000000c6106) #11 kudu::tablet::UpdateScanDeltaCompactionTest::CreateTable() /data1/kudu/src/kudu/integration-tests/update_scan_delta_compact-test.cc:81 (update_scan_delta_compact-test+0x0000000c1b6d) #12 kudu::tablet::UpdateScanDeltaCompactionTest_TestAll_Test::TestBody() /data1/kudu/src/kudu/integration-tests/update_scan_delta_compact-test.cc:169 (update_scan_delta_compact-test+0x0000000bacfe) #13 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /data1/kudu/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2078 (libgmock.so+0x000000044839) #14 main /data1/kudu/src/kudu/util/test_main.cc:45 (libkudu_test_main.so+0x000000001f54) SUMMARY: ThreadSanitizer: data race /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/generated_message_reflection.cc:1531 in google::protobuf::internal::GeneratedMessageReflection::HasBit(google::protobuf::Message const&, google::protobuf::FieldDescriptor const*) const ================== ================== WARNING: ThreadSanitizer: data race (pid=30436) Read of size 8 at 0x7d44000bde38 by thread T131 (mutexes: write M97279): #0 google::protobuf::internal::GeneratedMessageReflection::GetInt64(google::protobuf::Message const&, google::protobuf::FieldDescriptor const*) const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/generated_message_reflection.cc:978 (libprotobuf.so.9+0x000000105776) #1 google::protobuf::TextFormat::Printer::PrintFieldValue(google::protobuf::Message const&, google::protobuf::Reflection const*, google::protobuf::FieldDescriptor const*, int, google::protobuf::TextFormat::Printer::TextGenerator&) const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/text_format.cc:1562 (libprotobuf.so.9+0x00000011cac0) #2 google::protobuf::TextFormat::Printer::PrintField(google::protobuf::Message const&, google::protobuf::Reflection const*, google::protobuf::FieldDescriptor const*, google::protobuf::TextFormat::Printer::TextGenerator&) const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/text_format.cc:1493 (libprotobuf.so.9+0x00000011bf3b) #3 google::protobuf::TextFormat::Printer::Print(google::protobuf::Message const&, google::protobuf::TextFormat::Printer::TextGenerator&) const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/text_format.cc:1427 (libprotobuf.so.9+0x00000011aad0) #4 google::protobuf::TextFormat::Printer::Print(google::protobuf::Message const&, google::protobuf::io::ZeroCopyOutputStream*) const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/text_format.cc:1391 (libprotobuf.so.9+0x000000118274) #5 google::protobuf::Message::ShortDebugString() const /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/text_format.cc:88 (libprotobuf.so.9+0x000000118076) #6 kudu::tablet::WriteTransactionState::ToString() const /data1/kudu/src/kudu/tablet/transactions/write_transaction.cc:366 (libtablet.so+0x0000002e34a9) #7 kudu::tablet::WriteTransaction::ToString() const /data1/kudu/src/kudu/tablet/transactions/write_transaction.cc:203 (libtablet.so+0x0000002e2285) #8 kudu::tablet::TransactionDriver::ToStringUnlocked() const /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:118 (libtablet.so+0x0000002cd039) #9 kudu::tablet::TransactionDriver::ToString() const /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:112 (libtablet.so+0x0000002ccf27) #10 kudu::tablet::TabletPeer::GetInFlightTransactions(kudu::tablet::Transaction::TraceType, std::vector >*) const /data1/kudu/src/kudu/tablet/tablet_peer.cc:410 (libtablet.so+0x0000002b9e0e) #11 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) /data1/kudu/src/kudu/tserver/tserver-path-handlers.cc:130 (libtserver.so+0x0000001f092e) #12 boost::_mfi::mf2, std::allocator >*>::operator()(kudu::tserver::TabletServerPathHandlers*, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) const /usr/include/boost/bind/mem_fn_template.hpp:274 (libtserver.so+0x0000001fa47b) #13 void boost::_bi::list3, boost::arg<1>, boost::arg<2> >::operator(), std::allocator >*>, boost::_bi::list2, std::allocator >*&> >(boost::_bi::type, boost::_mfi::mf2, std::allocator >*>&, boost::_bi::list2, std::allocator >*&>&, int) /usr/include/boost/bind/bind.hpp:385 (libtserver.so+0x0000001fa261) #14 void boost::_bi::bind_t, std::allocator >*>, boost::_bi::list3, boost::arg<1>, boost::arg<2> > >::operator(), std::allocator >*>(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*&) /usr/include/boost/bind/bind_template.hpp:76 (libtserver.so+0x0000001fa11a) #15 boost::detail::function::void_function_obj_invoker2, std::allocator >*>, boost::_bi::list3, boost::arg<1>, boost::arg<2> > >, void, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*>::invoke(boost::detail::function::function_buffer&, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) /usr/include/boost/function/function_template.hpp:153 (libtserver.so+0x0000001f9dac) #16 boost::function2, std::allocator >*>::operator()(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) const /usr/include/boost/function/function_template.hpp:1012 (libserver_process.so+0x0000000bb78a) #17 kudu::Webserver::RunPathHandler(kudu::Webserver::PathHandler const&, sq_connection*, sq_request_info*) /data1/kudu/src/kudu/server/webserver.cc:354 (libserver_process.so+0x0000000b9027) #18 kudu::Webserver::BeginRequestCallback(sq_connection*, sq_request_info*) /data1/kudu/src/kudu/server/webserver.cc:298 (libserver_process.so+0x0000000b8637) #19 kudu::Webserver::BeginRequestCallbackStatic(sq_connection*) /data1/kudu/src/kudu/server/webserver.cc:273 (libserver_process.so+0x0000000b7dbd) #20 handle_request squeasel.c (libserver_process.so+0x0000000d64ff) Previous write of size 8 at 0x7d44000bde38 by thread T121 (mutexes: write M97294): #0 memset /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:586 (update_scan_delta_compact-test+0x00000005c43b) #1 kudu::consensus::OpId::Clear() /data1/kudu/src/kudu/consensus/opid.pb.cc:173 (libconsensus_metadata_proto.so+0x00000001dca7) #2 kudu::consensus::OpId::CopyFrom(kudu::consensus::OpId const&) /data1/kudu/src/kudu/consensus/opid.pb.cc:347 (libconsensus_metadata_proto.so+0x00000001ee53) #3 kudu::tablet::TransactionDriver::ReplicationFinished(kudu::Status const&) /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:272 (libtablet.so+0x0000002cc88f) #4 kudu::internal::RunnableAdapter::Run(kudu::tablet::TransactionDriver*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:190 (libtablet.so+0x0000002c569b) #5 kudu::internal::InvokeHelper, void (kudu::tablet::TransactionDriver*, kudu::Status const&)>::MakeItSo(kudu::internal::RunnableAdapter, kudu::tablet::TransactionDriver*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:887 (libtablet.so+0x0000002c54db) #6 kudu::internal::Invoker<1, kudu::internal::BindState, void (kudu::tablet::TransactionDriver*, kudu::Status const&), void (kudu::internal::UnretainedWrapper)>, void (kudu::tablet::TransactionDriver*, kudu::Status const&)>::Run(kudu::internal::BindStateBase*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:1116 (libtablet.so+0x0000002c53f5) #7 kudu::Callback::Run(kudu::Status const&) const /data1/kudu/src/kudu/gutil/callback.h:436 (libconsensus.so+0x0000000aa928) #8 kudu::consensus::ConsensusRound::NotifyReplicationFinished(kudu::Status const&) /data1/kudu/src/kudu/consensus/consensus.cc:59 (libconsensus.so+0x0000000a96f8) #9 kudu::internal::RunnableAdapter::Run(kudu::consensus::ConsensusRound*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:190 (libconsensus.so+0x00000010c8db) #10 kudu::internal::InvokeHelper, void (kudu::consensus::ConsensusRound*, kudu::Status const&)>::MakeItSo(kudu::internal::RunnableAdapter, kudu::consensus::ConsensusRound*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:887 (libconsensus.so+0x00000010c75b) #11 kudu::internal::Invoker<1, kudu::internal::BindState, void (kudu::consensus::ConsensusRound*, kudu::Status const&), void (scoped_refptr)>, void (kudu::consensus::ConsensusRound*, kudu::Status const&)>::Run(kudu::internal::BindStateBase*, kudu::Status const&) /data1/kudu/src/kudu/gutil/bind_internal.h:1116 (libconsensus.so+0x00000010c677) #12 kudu::Callback::Run(kudu::Status const&) const /data1/kudu/src/kudu/gutil/callback.h:436 (libconsensus.so+0x0000000aa928) #13 kudu::log::Log::AppendThread::RunThread() /data1/kudu/src/kudu/consensus/log.cc:218 (liblog.so+0x00000007fa51) #14 boost::_mfi::mf0::operator()(kudu::log::Log::AppendThread*) const /usr/include/boost/bind/mem_fn_template.hpp:49 (liblog.so+0x0000000a257b) #15 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:246 (liblog.so+0x0000000a2458) #16 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (liblog.so+0x0000000a238a) #17 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (liblog.so+0x0000000a1f80) #18 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #19 kudu::Thread::SuperviseThread(void*) /data1/kudu/src/kudu/util/thread.cc:507 (libkudu_util.so+0x0000002d8819) Location is heap block of size 288 at 0x7d44000bdd80 allocated by thread T60: #0 operator new(unsigned long) /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:37 (update_scan_delta_compact-test+0x0000000b8453) #1 kudu::tserver::TabletServiceImpl::Write(kudu::tserver::WriteRequestPB const*, kudu::tserver::WriteResponsePB*, kudu::rpc::RpcContext*) /data1/kudu/src/kudu/tserver/tablet_service.cc:711 (libtserver.so+0x000000192a83) #2 kudu::tserver::TabletServerServiceIf::Handle(kudu::rpc::InboundCall*) /data1/kudu/src/kudu/tserver/tserver_service.service.cc:83 (libtserver_service_proto.so+0x00000001b1f8) #3 kudu::rpc::ServicePool::RunThread() /data1/kudu/src/kudu/rpc/service_pool.cc:172 (libkrpc.so+0x0000001365a3) #4 boost::_mfi::mf0::operator()(kudu::rpc::ServicePool*) const /usr/include/boost/bind/mem_fn_template.hpp:49 (libkrpc.so+0x000000139e3b) #5 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:246 (libkrpc.so+0x000000139d18) #6 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkrpc.so+0x000000139c7a) #7 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkrpc.so+0x000000139930) #8 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #9 kudu::Thread::SuperviseThread(void*) /data1/kudu/src/kudu/util/thread.cc:507 (libkudu_util.so+0x0000002d8819) Mutex M97279 (0x7d280006df00) created at: #0 __tsan_atomic32_compare_exchange_strong /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:779 (update_scan_delta_compact-test+0x00000007d488) #1 base::subtle::Acquire_CompareAndSwap(int volatile*, int, int) /data1/kudu/src/kudu/gutil/atomicops-internals-tsan.h:85 (libmaster.so+0x000000116375) #2 base::SpinLock::Lock() /data1/kudu/src/kudu/gutil/spinlock.h:73 (libmaster.so+0x00000015c033) #3 kudu::simple_spinlock::lock() /data1/kudu/src/kudu/util/locks.h:42 (libmaster.so+0x00000015bfe8) #4 lock_guard /usr/include/boost/thread/locks.hpp:194 (libmaster.so+0x000000116f55) #5 kudu::tablet::TransactionDriver::PrepareAndStart() /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:175 (libtablet.so+0x0000002cf511) #6 kudu::tablet::TransactionDriver::PrepareAndStartTask() /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:153 (libtablet.so+0x0000002ce304) #7 kudu::internal::RunnableAdapter::Run(kudu::tablet::TransactionDriver*) /data1/kudu/src/kudu/gutil/bind_internal.h:134 (libtablet.so+0x0000002d47cb) #8 kudu::internal::InvokeHelper, void (kudu::tablet::TransactionDriver*)>::MakeItSo(kudu::internal::RunnableAdapter, kudu::tablet::TransactionDriver*) /data1/kudu/src/kudu/gutil/bind_internal.h:871 (libtablet.so+0x0000002d46fb) #9 kudu::internal::Invoker<1, kudu::internal::BindState, void (kudu::tablet::TransactionDriver*), void (kudu::internal::UnretainedWrapper)>, void (kudu::tablet::TransactionDriver*)>::Run(kudu::internal::BindStateBase*) /data1/kudu/src/kudu/gutil/bind_internal.h:1063 (libtablet.so+0x0000002d4682) #10 kudu::Callback::Run() const /data1/kudu/src/kudu/gutil/callback.h:396 (libconsensus.so+0x0000000f8efb) #11 boost::_mfi::cmf0 >::operator()(kudu::Callback const&) const /usr/include/boost/bind/mem_fn_template.hpp:117 (libconsensus.so+0x0000001046cb) #12 void boost::_bi::list1 > >::operator() >, boost::_bi::list0>(boost::_bi::type, boost::_mfi::cmf0 >&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:246 (libconsensus.so+0x0000001045a5) #13 boost::_bi::bind_t >, boost::_bi::list1 > > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libconsensus.so+0x00000010451a) #14 boost::detail::function::void_function_obj_invoker0 >, boost::_bi::list1 > > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libconsensus.so+0x000000104056) #15 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #16 kudu::FunctionRunnable::Run() /data1/kudu/src/kudu/util/threadpool.cc:46 (libkudu_util.so+0x0000002eb9dc) #17 kudu::ThreadPool::DispatchThread(bool) /data1/kudu/src/kudu/util/threadpool.cc:317 (libkudu_util.so+0x0000002e9b91) #18 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:162 (libkudu_util.so+0x0000002ef171) #19 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306 (libkudu_util.so+0x0000002ef03f) #20 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x0000002eef6a) #21 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x0000002eeae6) #22 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #23 kudu::Thread::SuperviseThread(void*) /data1/kudu/src/kudu/util/thread.cc:507 (libkudu_util.so+0x0000002d8819) Mutex M97294 (0x7d280006df30) created at: #0 __tsan_atomic32_compare_exchange_strong /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:779 (update_scan_delta_compact-test+0x00000007d488) #1 base::subtle::Acquire_CompareAndSwap(int volatile*, int, int) /data1/kudu/src/kudu/gutil/atomicops-internals-tsan.h:85 (libmaster.so+0x000000116375) #2 base::SpinLock::Lock() /data1/kudu/src/kudu/gutil/spinlock.h:73 (libmaster.so+0x00000015c033) #3 kudu::simple_spinlock::lock() /data1/kudu/src/kudu/util/locks.h:42 (libmaster.so+0x00000015bfe8) #4 lock_guard /usr/include/boost/thread/locks.hpp:194 (libmaster.so+0x000000116f55) #5 kudu::tablet::TransactionDriver::GetOpId() /data1/kudu/src/kudu/tablet/transactions/transaction_driver.cc:94 (libtablet.so+0x0000002ccd8a) #6 kudu::tablet::TabletPeer::GetInFlightTransactions(kudu::tablet::Transaction::TraceType, std::vector >*) const /data1/kudu/src/kudu/tablet/tablet_peer.cc:401 (libtablet.so+0x0000002b9d06) #7 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) /data1/kudu/src/kudu/tserver/tserver-path-handlers.cc:130 (libtserver.so+0x0000001f092e) #8 boost::_mfi::mf2, std::allocator >*>::operator()(kudu::tserver::TabletServerPathHandlers*, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) const /usr/include/boost/bind/mem_fn_template.hpp:274 (libtserver.so+0x0000001fa47b) #9 void boost::_bi::list3, boost::arg<1>, boost::arg<2> >::operator(), std::allocator >*>, boost::_bi::list2, std::allocator >*&> >(boost::_bi::type, boost::_mfi::mf2, std::allocator >*>&, boost::_bi::list2, std::allocator >*&>&, int) /usr/include/boost/bind/bind.hpp:385 (libtserver.so+0x0000001fa261) #10 void boost::_bi::bind_t, std::allocator >*>, boost::_bi::list3, boost::arg<1>, boost::arg<2> > >::operator(), std::allocator >*>(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*&) /usr/include/boost/bind/bind_template.hpp:76 (libtserver.so+0x0000001fa11a) #11 boost::detail::function::void_function_obj_invoker2, std::allocator >*>, boost::_bi::list3, boost::arg<1>, boost::arg<2> > >, void, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*>::invoke(boost::detail::function::function_buffer&, kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) /usr/include/boost/function/function_template.hpp:153 (libtserver.so+0x0000001f9dac) #12 boost::function2, std::allocator >*>::operator()(kudu::WebCallbackRegistry::WebRequest const&, std::basic_stringstream, std::allocator >*) const /usr/include/boost/function/function_template.hpp:1012 (libserver_process.so+0x0000000bb78a) #13 kudu::Webserver::RunPathHandler(kudu::Webserver::PathHandler const&, sq_connection*, sq_request_info*) /data1/kudu/src/kudu/server/webserver.cc:354 (libserver_process.so+0x0000000b9027) #14 kudu::Webserver::BeginRequestCallback(sq_connection*, sq_request_info*) /data1/kudu/src/kudu/server/webserver.cc:298 (libserver_process.so+0x0000000b8637) #15 kudu::Webserver::BeginRequestCallbackStatic(sq_connection*) /data1/kudu/src/kudu/server/webserver.cc:273 (libserver_process.so+0x0000000b7dbd) #16 handle_request squeasel.c (libserver_process.so+0x0000000d64ff) Thread T131 'sq_worker' (tid=30604, running) created by thread T105 at: #0 pthread_create /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:849 (update_scan_delta_compact-test+0x000000059fd3) #1 master_thread squeasel.c (libserver_process.so+0x0000000d3b51) Thread T121 'appender-30575' (tid=30575, running) created by thread T118 at: #0 pthread_create /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:849 (update_scan_delta_compact-test+0x000000059fd3) #1 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /data1/kudu/src/kudu/util/thread.cc:435 (libkudu_util.so+0x0000002d7dbc) #2 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::log::Log::AppendThread::* const&)(), kudu::log::Log::AppendThread* const&, scoped_refptr*) /data1/kudu/src/kudu/util/thread.h:137 (liblog.so+0x0000000900c6) #3 kudu::log::Log::AppendThread::Init() /data1/kudu/src/kudu/consensus/log.cc:150 (liblog.so+0x00000007e175) #4 kudu::log::Log::Init() /data1/kudu/src/kudu/consensus/log.cc:341 (liblog.so+0x0000000846ed) #5 kudu::log::Log::Open(kudu::log::LogOptions const&, kudu::FsManager*, std::string const&, kudu::Schema const&, unsigned int, scoped_refptr const&, scoped_refptr*) /data1/kudu/src/kudu/consensus/log.cc:271 (liblog.so+0x0000000837d5) #6 kudu::tablet::TabletBootstrap::OpenNewLog() /data1/kudu/src/kudu/tablet/tablet_bootstrap.cc:639 (libtablet.so+0x000000289f91) #7 kudu::tablet::TabletBootstrap::Bootstrap(std::shared_ptr*, scoped_refptr*, kudu::consensus::ConsensusBootstrapInfo*) /data1/kudu/src/kudu/tablet/tablet_bootstrap.cc:462 (libtablet.so+0x0000002867e8) #8 kudu::tablet::BootstrapTablet(scoped_refptr const&, scoped_refptr const&, std::shared_ptr const&, kudu::MetricRegistry*, kudu::tablet::TabletStatusListener*, std::shared_ptr*, scoped_refptr*, scoped_refptr const&, kudu::consensus::ConsensusBootstrapInfo*) /data1/kudu/src/kudu/tablet/tablet_bootstrap.cc:377 (libtablet.so+0x0000002852fc) #9 kudu::tserver::TSTabletManager::OpenTablet(scoped_refptr const&, scoped_refptr const&) /data1/kudu/src/kudu/tserver/ts_tablet_manager.cc:611 (libtserver.so+0x0000001c9450) #10 boost::_mfi::mf2 const&, scoped_refptr const&>::operator()(kudu::tserver::TSTabletManager*, scoped_refptr const&, scoped_refptr const&) const /usr/include/boost/bind/mem_fn_template.hpp:274 (libtserver.so+0x0000001e0b8b) #11 void boost::_bi::list3, boost::_bi::value >, boost::_bi::value > >::operator() const&, scoped_refptr const&>, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2 const&, scoped_refptr const&>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:385 (libtserver.so+0x0000001e0a53) #12 boost::_bi::bind_t const&, scoped_refptr const&>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libtserver.so+0x0000001e097a) #13 boost::detail::function::void_function_obj_invoker0 const&, scoped_refptr const&>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value > > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libtserver.so+0x0000001e04b6) #14 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #15 kudu::FunctionRunnable::Run() /data1/kudu/src/kudu/util/threadpool.cc:46 (libkudu_util.so+0x0000002eb9dc) #16 kudu::ThreadPool::DispatchThread(bool) /data1/kudu/src/kudu/util/threadpool.cc:317 (libkudu_util.so+0x0000002e9b91) #17 boost::_mfi::mf1::operator()(kudu::ThreadPool*, bool) const /usr/include/boost/bind/mem_fn_template.hpp:162 (libkudu_util.so+0x0000002ef171) #18 void boost::_bi::list2, boost::_bi::value >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf1&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306 (libkudu_util.so+0x0000002ef03f) #19 boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 (libkudu_util.so+0x0000002eef6a) #20 boost::detail::function::void_function_obj_invoker0, boost::_bi::list2, boost::_bi::value > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153 (libkudu_util.so+0x0000002eeae6) #21 boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 (libkrpc.so+0x0000000b259a) #22 kudu::Thread::SuperviseThread(void*) /data1/kudu/src/kudu/util/thread.cc:507 (libkudu_util.so+0x0000002d8819) Thread T60 'rpc worker-3050' (tid=30501, running) created by main thread at: #0 pthread_create /data1/kudu/thirdparty/llvm-3.7.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:849 (update_scan_delta_compact-test+0x000000059fd3) #1 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function const&, scoped_refptr*) /data1/kudu/src/kudu/util/thread.cc:435 (libkudu_util.so+0x0000002d7dbc) #2 kudu::Status kudu::Thread::Create(std::string const&, std::string const&, void (kudu::rpc::ServicePool::* const&)(), kudu::rpc::ServicePool* const&, scoped_refptr*) /data1/kudu/src/kudu/util/thread.h:137 (libkrpc.so+0x000000137c26) #3 kudu::rpc::ServicePool::Init(int) /data1/kudu/src/kudu/rpc/service_pool.cc:76 (libkrpc.so+0x000000135ce4) #4 kudu::RpcServer::RegisterService(gscoped_ptr >) /data1/kudu/src/kudu/server/rpc_server.cc:122 (libserver_process.so+0x0000000898c8) #5 kudu::server::ServerBase::RegisterService(gscoped_ptr >) /data1/kudu/src/kudu/server/server_base.cc:237 (libserver_process.so+0x000000095c7e) #6 kudu::tserver::TabletServer::Start() /data1/kudu/src/kudu/tserver/tablet_server.cc:117 (libtserver.so+0x000000188726) #7 kudu::tserver::MiniTabletServer::Start() /data1/kudu/src/kudu/tserver/mini_tablet_server.cc:77 (libtserver.so+0x00000011b747) #8 kudu::MiniCluster::AddTabletServer() /data1/kudu/src/kudu/integration-tests/mini_cluster.cc:173 (libintegration-tests.so+0x0000000d5401) #9 kudu::MiniCluster::Start() /data1/kudu/src/kudu/integration-tests/mini_cluster.cc:89 (libintegration-tests.so+0x0000000d36bb) #10 kudu::tablet::UpdateScanDeltaCompactionTest::InitCluster() /data1/kudu/src/kudu/integration-tests/update_scan_delta_compact-test.cc:113 (update_scan_delta_compact-test+0x0000000c6106) #11 kudu::tablet::UpdateScanDeltaCompactionTest::CreateTable() /data1/kudu/src/kudu/integration-tests/update_scan_delta_compact-test.cc:81 (update_scan_delta_compact-test+0x0000000c1b6d) #12 kudu::tablet::UpdateScanDeltaCompactionTest_TestAll_Test::TestBody() /data1/kudu/src/kudu/integration-tests/update_scan_delta_compact-test.cc:169 (update_scan_delta_compact-test+0x0000000bacfe) #13 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /data1/kudu/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2078 (libgmock.so+0x000000044839) #14 main /data1/kudu/src/kudu/util/test_main.cc:45 (libkudu_test_main.so+0x000000001f54) SUMMARY: ThreadSanitizer: data race /data1/kudu/thirdparty/protobuf-2.6.1/src/google/protobuf/generated_message_reflection.cc:978 in google::protobuf::internal::GeneratedMessageReflection::GetInt64(google::protobuf::Message const&, google::protobuf::FieldDescriptor const*) const ================== W1216 18:23:17.468617 30564 kernel_stack_watchdog.cc:122] Thread 30575 stuck at src/kudu/consensus/log.cc:214 for 184ms: Kernel stack: [] futex_wait_queue_me+0xba/0xf0 [] futex_wait+0x1c0/0x310 [] do_futex+0x121/0xae0 [] sys_futex+0x7b/0x170 [] system_call_fastpath+0x16/0x1b [] 0xffffffffffffffff User stack: @ 0x7f443946a7e1 __tsan::ProcessPendingSignals() at ??:0 @ 0x7f44394abdc5 __tsan_atomic32_load at ??:0 @ 0x7f4438dd9aeb base::subtle::NoBarrier_Load() at ??:0 @ 0x7f442ecd9d13 base::SpinLock::SpinLoop() at ??:0 @ 0x7f442ecd9e9a base::SpinLock::SlowLock() at ??:0 @ 0x7f4438407046 base::SpinLock::Lock() at ??:0 @ 0x7f4438406fe9 kudu::simple_spinlock::lock() at ??:0 @ 0x7f44383c1f56 boost::lock_guard<>::lock_guard() at ??:0 @ 0x7f44379a08b5 kudu::tablet::TransactionDriver::ReplicationFinished() at ??:0 @ 0x7f443799969c kudu::internal::RunnableAdapter<>::Run() at ??:0 @ 0x7f44379994dc kudu::internal::InvokeHelper<>::MakeItSo() at ??:0 @ 0x7f44379993f6 kudu::internal::Invoker<>::Run() at ??:0 @ 0x7f44373d1929 kudu::Callback<>::Run() at ??:0 @ 0x7f44373d06f9 kudu::consensus::ConsensusRound::NotifyReplicationFinished() at ??:0 @ 0x7f44374338dc kudu::internal::RunnableAdapter<>::Run() at ??:0 @ 0x7f443743375c kudu::internal::InvokeHelper<>::MakeItSo() at ??:0 I1216 18:23:17.507807 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.164s user 0.005s sys 0.002s I1216 18:23:17.660068 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.152s user 0.002s sys 0.003s I1216 18:23:17.859319 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.199s user 0.003s sys 0.001s I1216 18:23:18.023705 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.164s user 0.002s sys 0.003s I1216 18:23:18.077513 30599 update_scan_delta_compact-test.cc:243] Time spent Update: real 1.278s user 0.594s sys 0.114s W1216 18:23:18.111403 30566 outbound_call.cc:185] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.0.1:36395, user_credentials={real_user=dan, eff_user=}} blocked reactor thread for 39013.7us I1216 18:23:18.193111 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.169s user 0.001s sys 0.003s I1216 18:23:18.436208 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.243s user 0.002s sys 0.002s I1216 18:23:18.596982 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.160s user 0.002s sys 0.003s I1216 18:23:18.800792 30599 update_scan_delta_compact-test.cc:243] Time spent Update: real 0.723s user 0.487s sys 0.037s I1216 18:23:18.840689 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.243s user 0.002s sys 0.003s I1216 18:23:18.965497 30599 update_scan_delta_compact-test.cc:243] Time spent Update: real 0.164s user 0.083s sys 0.011s I1216 18:23:19.074270 30600 update_scan_delta_compact-test.cc:259] Time spent Scan: real 0.233s user 0.003s sys 0.001s I1216 18:23:19.076495 30436 tablet_server.cc:132] TabletServer shutting down... I1216 18:23:19.165701 30436 ts_tablet_manager.cc:680] Shutting down tablet manager... I1216 18:23:19.166126 30436 tablet_peer.cc:215] Initiating TabletPeer shutdown for tablet: 7b5b4651c246417b8a22a497c0eb9fdb I1216 18:23:19.166756 30436 maintenance_manager.cc:182] Unregistered op CompactRowSetsOp(7b5b4651c246417b8a22a497c0eb9fdb) I1216 18:23:19.167081 30436 maintenance_manager.cc:182] Unregistered op MinorDeltaCompactionOp(7b5b4651c246417b8a22a497c0eb9fdb) I1216 18:23:19.167343 30436 maintenance_manager.cc:182] Unregistered op MajorDeltaCompactionOp(7b5b4651c246417b8a22a497c0eb9fdb) I1216 18:23:19.167706 30436 maintenance_manager.cc:182] Unregistered op FlushMRSOp(7b5b4651c246417b8a22a497c0eb9fdb) I1216 18:23:19.168014 30436 maintenance_manager.cc:182] Unregistered op FlushDeltaMemStoresOp(7b5b4651c246417b8a22a497c0eb9fdb) I1216 18:23:19.168261 30436 maintenance_manager.cc:182] Unregistered op LogGCOp(7b5b4651c246417b8a22a497c0eb9fdb) I1216 18:23:19.196112 30436 tablet_server.cc:141] TabletServer shut down complete. Bye! I1216 18:23:19.196293 30436 tablet_server.cc:132] TabletServer shutting down... I1216 18:23:19.196701 30436 tablet_server.cc:141] TabletServer shut down complete. Bye! I1216 18:23:19.204697 30436 master.cc:169] Master@127.0.0.1:46884 shutting down... I1216 18:23:19.306861 30436 tablet_peer.cc:215] Initiating TabletPeer shutdown for tablet: 00000000000000000000000000000000 I1216 18:23:19.307317 30436 maintenance_manager.cc:182] Unregistered op CompactRowSetsOp(00000000000000000000000000000000) I1216 18:23:19.307615 30436 maintenance_manager.cc:182] Unregistered op MinorDeltaCompactionOp(00000000000000000000000000000000) I1216 18:23:19.307880 30436 maintenance_manager.cc:182] Unregistered op MajorDeltaCompactionOp(00000000000000000000000000000000) I1216 18:23:19.308199 30436 maintenance_manager.cc:182] Unregistered op FlushMRSOp(00000000000000000000000000000000) I1216 18:23:19.308447 30436 maintenance_manager.cc:182] Unregistered op FlushDeltaMemStoresOp(00000000000000000000000000000000) I1216 18:23:19.308691 30436 maintenance_manager.cc:182] Unregistered op LogGCOp(00000000000000000000000000000000) I1216 18:23:19.312968 30436 master.cc:173] Master@127.0.0.1:46884 shutdown complete. [ OK ] UpdateScanDeltaCompactionTest.TestAll (7770 ms) [----------] 1 test from UpdateScanDeltaCompactionTest (7770 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (7771 ms total) [ PASSED ] 1 test. ThreadSanitizer: reported 2 warnings