[==========] Running 15 tests from 3 test cases. [----------] Global test environment set-up. [----------] 10 tests from DeleteTableTest [ RUN ] DeleteTableTest.TestDeleteEmptyTable I0628 23:54:42.822856 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:42.928077 27219 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:42.928555 27219 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:42.940600 27219 master_main.cc:58] Initializing master server... I0628 23:54:42.942675 27219 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 127351 I0628 23:54:42.949270 27219 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:54:42.949518 27219 server_base.cc:165] Creating new FS layout I0628 23:54:42.964187 27219 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/master/instance: uuid: "b92e763b8ad549259a2b5be232a28a83" format_stamp: "Formatted at 2016-06-28 23:54:42 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:42.977125 27219 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/master uuid: "b92e763b8ad549259a2b5be232a28a83" format_stamp: "Formatted at 2016-06-28 23:54:42 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:43.011714 27219 master_main.cc:61] Starting Master server... I0628 23:54:43.047626 27219 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:43411 I0628 23:54:43.049201 27219 webserver.cc:122] Starting webserver on localhost:0 I0628 23:54:43.049331 27219 webserver.cc:133] Document root disabled I0628 23:54:43.050302 27219 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:54:43.051671 27219 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:33323/ I0628 23:54:43.053272 27219 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:54:43.059732 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 27219 I0628 23:54:43.060807 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43411 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:43.079949 27262 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83: Bootstrap starting. I0628 23:54:43.083894 27262 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:43.084554 27262 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83: No blocks or log segments found. Creating new log. I0628 23:54:43.086367 27262 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:43.090981 27262 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83: No bootstrap required, opened a new log I0628 23:54:43.097115 27262 raft_consensus.cc:269] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "b92e763b8ad549259a2b5be232a28a83" member_type: VOTER } I0628 23:54:43.098104 27262 raft_consensus.cc:301] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:43.098363 27262 raft_consensus.cc:499] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92e763b8ad549259a2b5be232a28a83, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:43.098742 27262 consensus_queue.cc:162] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:43.099066 27262 raft_consensus.cc:314] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:54:43.099243 27262 raft_consensus.cc:370] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:43.099400 27262 raft_consensus.cc:2019] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:43.100983 27262 raft_consensus.cc:1970] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.612s I0628 23:54:43.101212 27262 raft_consensus.cc:386] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "b92e763b8ad549259a2b5be232a28a83" member_type: VOTER } I0628 23:54:43.103126 27262 leader_election.cc:248] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:43.104876 27267 raft_consensus.cc:1970] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.632s I0628 23:54:43.105149 27267 raft_consensus.cc:1859] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:43.107600 27267 raft_consensus.cc:464] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 1 LEADER]: Becoming Leader. State: Replica: b92e763b8ad549259a2b5be232a28a83, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:43.108363 27267 consensus_queue.cc:145] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b92e763b8ad549259a2b5be232a28a83" member_type: VOTER } I0628 23:54:43.127616 27262 sys_catalog.cc:235] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "b92e763b8ad549259a2b5be232a28a83" config { opid_index: -1 peers { permanent_uuid: "b92e763b8ad549259a2b5be232a28a83" member_type: VOTER } } I0628 23:54:43.134421 27262 sys_catalog.cc:238] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [sys.catalog]: This master's current role is: LEADER I0628 23:54:43.127039 27268 sys_catalog.cc:235] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92e763b8ad549259a2b5be232a28a83" config { opid_index: -1 peers { permanent_uuid: "b92e763b8ad549259a2b5be232a28a83" member_type: VOTER } } I0628 23:54:43.128170 27269 sys_catalog.cc:235] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92e763b8ad549259a2b5be232a28a83. Latest consensus state: current_term: 1 leader_uuid: "b92e763b8ad549259a2b5be232a28a83" config { opid_index: -1 peers { permanent_uuid: "b92e763b8ad549259a2b5be232a28a83" member_type: VOTER } } I0628 23:54:43.135188 27269 sys_catalog.cc:238] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [sys.catalog]: This master's current role is: LEADER I0628 23:54:43.135170 27268 sys_catalog.cc:238] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [sys.catalog]: This master's current role is: LEADER I0628 23:54:43.137589 27271 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:54:43.139611 27272 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:54:43.140940 27262 sys_catalog.cc:311] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:54:43.151973 27219 master_main.cc:64] Master server successfully started. I0628 23:54:43.185910 27263 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:43.186386 27263 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:43.200816 27263 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:43.203637 27263 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 127851 I0628 23:54:43.210717 27263 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:54:43.210979 27263 server_base.cc:165] Creating new FS layout I0628 23:54:43.220293 27263 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "0e646222110f4bcc80e1f455170d0646" format_stamp: "Formatted at 2016-06-28 23:54:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:43.239449 27263 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-0 uuid: "0e646222110f4bcc80e1f455170d0646" format_stamp: "Formatted at 2016-06-28 23:54:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:43.292366 27263 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:43.403220 27263 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:47545 I0628 23:54:43.404718 27263 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:54:43.404805 27263 webserver.cc:133] Document root disabled I0628 23:54:43.406576 27263 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:51566/ I0628 23:54:43.407934 27263 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:54:43.410365 27263 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:43.418285 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 27263 I0628 23:54:43.419148 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43411 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:43.425637 27387 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43411 I0628 23:54:43.426034 27387 heartbeater.cc:319] Registering TS with master... I0628 23:54:43.426306 27387 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:43.427952 27239 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "0e646222110f4bcc80e1f455170d0646" instance_seqno: 1467158083383073 } with Master I0628 23:54:43.534024 27391 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:43.534512 27391 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:43.548382 27391 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:43.551187 27391 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 127851 I0628 23:54:43.557363 27391 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:54:43.557612 27391 server_base.cc:165] Creating new FS layout I0628 23:54:43.565876 27391 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "e77f738d190d43c6879ecc54f0fa8773" format_stamp: "Formatted at 2016-06-28 23:54:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:43.574367 27391 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-1 uuid: "e77f738d190d43c6879ecc54f0fa8773" format_stamp: "Formatted at 2016-06-28 23:54:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:43.610429 27391 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:43.713567 27391 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:58300 I0628 23:54:43.715066 27391 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:54:43.715180 27391 webserver.cc:133] Document root disabled I0628 23:54:43.716776 27391 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:45711/ I0628 23:54:43.718327 27391 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:54:43.721006 27391 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:43.725872 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 27391 I0628 23:54:43.726451 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43411 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:43.732699 27505 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43411 I0628 23:54:43.733124 27505 heartbeater.cc:319] Registering TS with master... I0628 23:54:43.733417 27505 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:43.734926 27239 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "e77f738d190d43c6879ecc54f0fa8773" instance_seqno: 1467158083692467 } with Master I0628 23:54:43.839365 27508 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:43.839799 27508 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:43.852597 27508 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:43.855350 27508 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 127851 I0628 23:54:43.861112 27508 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:54:43.861354 27508 server_base.cc:165] Creating new FS layout I0628 23:54:43.868921 27508 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "69c1c022bd0f4d5688848e82b8419a79" format_stamp: "Formatted at 2016-06-28 23:54:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:43.877081 27508 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-2 uuid: "69c1c022bd0f4d5688848e82b8419a79" format_stamp: "Formatted at 2016-06-28 23:54:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:43.909934 27508 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:44.001554 27508 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:37344 I0628 23:54:44.003413 27508 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:54:44.003509 27508 webserver.cc:133] Document root disabled I0628 23:54:44.005362 27508 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:58343/ I0628 23:54:44.006690 27508 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteEmptyTable.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:54:44.009464 27508 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:44.012404 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 27508 I0628 23:54:44.034543 27622 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43411 I0628 23:54:44.034837 27622 heartbeater.cc:319] Registering TS with master... I0628 23:54:44.035117 27622 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:44.036764 27239 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "69c1c022bd0f4d5688848e82b8419a79" instance_seqno: 1467158083982542 } with Master I0628 23:54:44.041939 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:54:44.097106 27239 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:54258: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:54:44.140476 27559 tablet_service.cc:612] Processing CreateTablet for tablet 9eb415a9935543e783ff9e9d0e0676be (table=test-workload [id=9ee66f95ad494f0bbf0c092cae84a52d]), partition=range: [(), ()) I0628 23:54:44.143543 27442 tablet_service.cc:612] Processing CreateTablet for tablet 9eb415a9935543e783ff9e9d0e0676be (table=test-workload [id=9ee66f95ad494f0bbf0c092cae84a52d]), partition=range: [(), ()) I0628 23:54:44.146322 27324 tablet_service.cc:612] Processing CreateTablet for tablet 9eb415a9935543e783ff9e9d0e0676be (table=test-workload [id=9ee66f95ad494f0bbf0c092cae84a52d]), partition=range: [(), ()) I0628 23:54:44.158188 27559 ts_tablet_manager.cc:760] Registered tablet 9eb415a9935543e783ff9e9d0e0676be I0628 23:54:44.166532 27442 ts_tablet_manager.cc:760] Registered tablet 9eb415a9935543e783ff9e9d0e0676be I0628 23:54:44.168501 27643 ts_tablet_manager.cc:628] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: Bootstrapping tablet I0628 23:54:44.168671 27324 ts_tablet_manager.cc:760] Registered tablet 9eb415a9935543e783ff9e9d0e0676be I0628 23:54:44.170048 27642 ts_tablet_manager.cc:628] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: Bootstrapping tablet I0628 23:54:44.173458 27642 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: Bootstrap starting. I0628 23:54:44.174289 27643 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: Bootstrap starting. I0628 23:54:44.178762 27642 tablet_bootstrap.cc:553] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:44.179857 27643 tablet_bootstrap.cc:553] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: Time spent opening tablet: real 0.002s user 0.004s sys 0.000s I0628 23:54:44.180438 27642 tablet_bootstrap.cc:493] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: No blocks or log segments found. Creating new log. I0628 23:54:44.180631 27643 tablet_bootstrap.cc:493] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: No blocks or log segments found. Creating new log. I0628 23:54:44.181576 27644 ts_tablet_manager.cc:628] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: Bootstrapping tablet I0628 23:54:44.182873 27642 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:44.183763 27643 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:44.184788 27644 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: Bootstrap starting. I0628 23:54:44.192325 27644 tablet_bootstrap.cc:553] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:54:44.192898 27644 tablet_bootstrap.cc:493] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: No blocks or log segments found. Creating new log. I0628 23:54:44.195104 27644 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:44.203428 27643 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: No bootstrap required, opened a new log I0628 23:54:44.203835 27643 ts_tablet_manager.cc:633] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.012s I0628 23:54:44.204219 27642 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: No bootstrap required, opened a new log I0628 23:54:44.204759 27642 ts_tablet_manager.cc:633] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.012s I0628 23:54:44.215020 27644 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: No bootstrap required, opened a new log I0628 23:54:44.214038 27642 raft_consensus.cc:269] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "69c1c022bd0f4d5688848e82b8419a79" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37344 } } peers { permanent_uuid: "0e646222110f4bcc80e1f455170d0646" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47545 } } peers { permanent_uuid: "e77f738d190d43c6879ecc54f0fa8773" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 58300 } } I0628 23:54:44.215484 27644 ts_tablet_manager.cc:633] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.016s I0628 23:54:44.216536 27642 raft_consensus.cc:301] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:44.215874 27643 raft_consensus.cc:269] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "69c1c022bd0f4d5688848e82b8419a79" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37344 } } peers { permanent_uuid: "0e646222110f4bcc80e1f455170d0646" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47545 } } peers { permanent_uuid: "e77f738d190d43c6879ecc54f0fa8773" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 58300 } } I0628 23:54:44.216922 27642 raft_consensus.cc:499] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c1c022bd0f4d5688848e82b8419a79, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:44.217550 27642 consensus_queue.cc:162] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:44.218003 27643 raft_consensus.cc:301] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:44.218329 27643 raft_consensus.cc:499] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77f738d190d43c6879ecc54f0fa8773, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:44.218955 27643 consensus_queue.cc:162] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:44.227584 27643 ts_tablet_manager.cc:659] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: Time spent starting tablet: real 0.023s user 0.012s sys 0.004s I0628 23:54:44.229120 27644 raft_consensus.cc:269] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "69c1c022bd0f4d5688848e82b8419a79" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37344 } } peers { permanent_uuid: "0e646222110f4bcc80e1f455170d0646" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47545 } } peers { permanent_uuid: "e77f738d190d43c6879ecc54f0fa8773" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 58300 } } I0628 23:54:44.230058 27642 ts_tablet_manager.cc:659] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: Time spent starting tablet: real 0.025s user 0.016s sys 0.000s I0628 23:54:44.230794 27644 raft_consensus.cc:301] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:44.231060 27644 raft_consensus.cc:499] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e646222110f4bcc80e1f455170d0646, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:44.231562 27644 consensus_queue.cc:162] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:44.234757 27644 ts_tablet_manager.cc:659] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: Time spent starting tablet: real 0.019s user 0.012s sys 0.004s I0628 23:54:44.417616 27652 raft_consensus.cc:370] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:44.417789 27652 raft_consensus.cc:2019] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:44.419977 27652 raft_consensus.cc:1970] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.590s I0628 23:54:44.420300 27652 raft_consensus.cc:386] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "69c1c022bd0f4d5688848e82b8419a79" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37344 } } peers { permanent_uuid: "0e646222110f4bcc80e1f455170d0646" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47545 } } peers { permanent_uuid: "e77f738d190d43c6879ecc54f0fa8773" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 58300 } } I0628 23:54:44.423856 27652 leader_election.cc:223] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [CANDIDATE]: Term 1 election: Requesting vote from peer 0e646222110f4bcc80e1f455170d0646 I0628 23:54:44.424325 27652 leader_election.cc:223] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [CANDIDATE]: Term 1 election: Requesting vote from peer e77f738d190d43c6879ecc54f0fa8773 I0628 23:54:44.429757 27344 raft_consensus.cc:2019] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:44.431547 27462 raft_consensus.cc:2019] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:44.431677 27344 raft_consensus.cc:1970] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.569s I0628 23:54:44.433281 27344 raft_consensus.cc:1677] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69c1c022bd0f4d5688848e82b8419a79 in term 1. I0628 23:54:44.433797 27462 raft_consensus.cc:1970] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.559s I0628 23:54:44.434339 27515 leader_election.cc:347] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [CANDIDATE]: Term 1 election: Vote granted by peer 0e646222110f4bcc80e1f455170d0646 I0628 23:54:44.434542 27515 leader_election.cc:248] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:44.435060 27654 raft_consensus.cc:1970] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.626s I0628 23:54:44.435287 27654 raft_consensus.cc:1859] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:44.435397 27462 raft_consensus.cc:1677] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69c1c022bd0f4d5688848e82b8419a79 in term 1. I0628 23:54:44.436197 27515 leader_election.cc:347] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [CANDIDATE]: Term 1 election: Vote granted by peer e77f738d190d43c6879ecc54f0fa8773 I0628 23:54:44.436890 27654 raft_consensus.cc:464] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 LEADER]: Becoming Leader. State: Replica: 69c1c022bd0f4d5688848e82b8419a79, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:44.437623 27654 consensus_queue.cc:145] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "69c1c022bd0f4d5688848e82b8419a79" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37344 } } peers { permanent_uuid: "0e646222110f4bcc80e1f455170d0646" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47545 } } peers { permanent_uuid: "e77f738d190d43c6879ecc54f0fa8773" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 58300 } } I0628 23:54:44.461850 27237 catalog_manager.cc:1673] Tablet: 9eb415a9935543e783ff9e9d0e0676be reported consensus state change. New consensus state: current_term: 1 leader_uuid: "69c1c022bd0f4d5688848e82b8419a79" config { opid_index: -1 peers { permanent_uuid: "69c1c022bd0f4d5688848e82b8419a79" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37344 } } peers { permanent_uuid: "0e646222110f4bcc80e1f455170d0646" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47545 } } peers { permanent_uuid: "e77f738d190d43c6879ecc54f0fa8773" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 58300 } } } W0628 23:54:44.462579 27662 raft_consensus_state.cc:541] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:54:44.534827 27237 catalog_manager.cc:1036] Servicing DeleteTable request from {real_user=slave, eff_user=} at 127.0.0.1:54257: table { table_name: "test-workload" } I0628 23:54:44.541127 27237 catalog_manager.cc:2531] Sending DeleteTablet for 3 replicas of tablet 9eb415a9935543e783ff9e9d0e0676be I0628 23:54:44.541242 27237 catalog_manager.cc:2547] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83: Deleting tablet 9eb415a9935543e783ff9e9d0e0676be on peer 69c1c022bd0f4d5688848e82b8419a79 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:44 UTC) I0628 23:54:44.541679 27237 catalog_manager.cc:2547] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83: Deleting tablet 9eb415a9935543e783ff9e9d0e0676be on peer 0e646222110f4bcc80e1f455170d0646 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:44 UTC) I0628 23:54:44.542120 27237 catalog_manager.cc:2547] T 00000000000000000000000000000000 P b92e763b8ad549259a2b5be232a28a83: Deleting tablet 9eb415a9935543e783ff9e9d0e0676be on peer e77f738d190d43c6879ecc54f0fa8773 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:44 UTC) I0628 23:54:44.542675 27559 tablet_service.cc:653] Processing DeleteTablet for tablet 9eb415a9935543e783ff9e9d0e0676be with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:44 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:57036 I0628 23:54:44.543121 27559 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9eb415a9935543e783ff9e9d0e0676be I0628 23:54:44.543200 27324 tablet_service.cc:653] Processing DeleteTablet for tablet 9eb415a9935543e783ff9e9d0e0676be with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:44 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:44281 I0628 23:54:44.543593 27559 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.543550 27442 tablet_service.cc:653] Processing DeleteTablet for tablet 9eb415a9935543e783ff9e9d0e0676be with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:44 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:57443 I0628 23:54:44.543664 27324 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9eb415a9935543e783ff9e9d0e0676be I0628 23:54:44.543912 27442 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9eb415a9935543e783ff9e9d0e0676be I0628 23:54:44.543917 27559 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544103 27324 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544176 27559 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544276 27442 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544441 27324 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544453 27559 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544517 27442 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544649 27559 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544683 27324 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544735 27442 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544848 27559 maintenance_manager.cc:180] Unregistered op LogGCOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.544955 27324 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.545002 27442 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.545088 27559 raft_consensus.cc:1495] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 LEADER]: Raft consensus shutting down. I0628 23:54:44.545200 27442 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.545207 27324 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.545372 27442 maintenance_manager.cc:180] Unregistered op LogGCOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.545383 27324 maintenance_manager.cc:180] Unregistered op LogGCOp(9eb415a9935543e783ff9e9d0e0676be) I0628 23:54:44.545562 27442 raft_consensus.cc:1495] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:54:44.545581 27324 raft_consensus.cc:1495] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:54:44.545929 27442 raft_consensus.cc:1511] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:54:44.546185 27324 raft_consensus.cc:1511] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:54:44.546391 27559 consensus_peers.cc:351] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 -> Peer e77f738d190d43c6879ecc54f0fa8773 (127.106.77.1:58300): Closing peer: e77f738d190d43c6879ecc54f0fa8773 I0628 23:54:44.546972 27559 consensus_peers.cc:351] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 -> Peer 0e646222110f4bcc80e1f455170d0646 (127.106.77.0:47545): Closing peer: 0e646222110f4bcc80e1f455170d0646 I0628 23:54:44.547891 27559 raft_consensus_state.cc:377] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 LEADER]: Trying to abort 1 pending transactions. I0628 23:54:44.548101 27559 raft_consensus_state.cc:382] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 LEADER]: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6009479513871589376 op_type: NO_OP noop_request { } I0628 23:54:44.548619 27559 raft_consensus.cc:1906] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: NO_OP replication failed: Aborted: Transaction aborted I0628 23:54:44.548785 27559 raft_consensus.cc:1511] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79 [term 1 LEADER]: Raft consensus is shut down! I0628 23:54:44.552497 27442 ts_tablet_manager.cc:955] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:54:44.552673 27324 ts_tablet_manager.cc:955] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:54:44.555652 27559 ts_tablet_manager.cc:955] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:54:44.556152 27237 catalog_manager.cc:1119] Successfully deleted table test-workload [id=9ee66f95ad494f0bbf0c092cae84a52d] per request from {real_user=slave, eff_user=} at 127.0.0.1:54257 I0628 23:54:44.557070 27324 ts_tablet_manager.cc:965] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: Tablet deleted. Last logged OpId: 0.0 I0628 23:54:44.558416 27324 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P 0e646222110f4bcc80e1f455170d0646: Deleted tablet blocks from disk I0628 23:54:44.560065 27442 ts_tablet_manager.cc:965] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: Tablet deleted. Last logged OpId: 0.0 I0628 23:54:44.560389 27645 catalog_manager.cc:2226] TS 0e646222110f4bcc80e1f455170d0646: tablet 9eb415a9935543e783ff9e9d0e0676be (table test-workload [id=9ee66f95ad494f0bbf0c092cae84a52d]) successfully deleted I0628 23:54:44.561383 27442 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P e77f738d190d43c6879ecc54f0fa8773: Deleted tablet blocks from disk I0628 23:54:44.562479 27559 ts_tablet_manager.cc:965] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: Tablet deleted. Last logged OpId: 1.1 I0628 23:54:44.563611 27559 tablet_bootstrap.cc:393] T 9eb415a9935543e783ff9e9d0e0676be P 69c1c022bd0f4d5688848e82b8419a79: Deleted tablet blocks from disk I0628 23:54:44.563572 27645 catalog_manager.cc:2226] TS e77f738d190d43c6879ecc54f0fa8773: tablet 9eb415a9935543e783ff9e9d0e0676be (table test-workload [id=9ee66f95ad494f0bbf0c092cae84a52d]) successfully deleted I0628 23:54:44.565294 27645 catalog_manager.cc:2226] TS 69c1c022bd0f4d5688848e82b8419a79: tablet 9eb415a9935543e783ff9e9d0e0676be (table test-workload [id=9ee66f95ad494f0bbf0c092cae84a52d]) successfully deleted W0628 23:54:44.573061 27629 client-internal.cc:743] GetTableSchemaRpc(table_name: test-workload, num_attempts: 1) failed: Not found: The table does not exist: table_name: "test-workload" I0628 23:54:44.603359 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 27219 I0628 23:54:44.619459 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 27263 I0628 23:54:44.634227 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 27391 I0628 23:54:44.648423 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 27508 [ OK ] DeleteTableTest.TestDeleteEmptyTable (1882 ms) [ RUN ] DeleteTableTest.TestDeleteTableDestUuidValidation I0628 23:54:44.672302 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:44.782877 27665 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:44.783432 27665 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:44.795411 27665 master_main.cc:58] Initializing master server... I0628 23:54:44.797449 27665 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 128351 I0628 23:54:44.804200 27665 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:54:44.804411 27665 server_base.cc:165] Creating new FS layout I0628 23:54:44.817229 27665 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/master/instance: uuid: "b06d8ce417fb408f8c07252e9a5533b0" format_stamp: "Formatted at 2016-06-28 23:54:44 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:44.830453 27665 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/master uuid: "b06d8ce417fb408f8c07252e9a5533b0" format_stamp: "Formatted at 2016-06-28 23:54:44 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:44.863059 27665 master_main.cc:61] Starting Master server... I0628 23:54:44.897107 27665 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:39993 I0628 23:54:44.898417 27665 webserver.cc:122] Starting webserver on localhost:0 I0628 23:54:44.898521 27665 webserver.cc:133] Document root disabled I0628 23:54:44.899238 27665 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:54:44.900475 27665 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:35242/ I0628 23:54:44.901991 27665 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:54:44.908619 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 27665 I0628 23:54:44.909435 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:39993 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:44.928091 27708 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0: Bootstrap starting. I0628 23:54:44.931893 27708 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:44.932504 27708 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0: No blocks or log segments found. Creating new log. I0628 23:54:44.934116 27708 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:44.938772 27708 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0: No bootstrap required, opened a new log I0628 23:54:44.944299 27708 raft_consensus.cc:269] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "b06d8ce417fb408f8c07252e9a5533b0" member_type: VOTER } I0628 23:54:44.945236 27708 raft_consensus.cc:301] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:44.945479 27708 raft_consensus.cc:499] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06d8ce417fb408f8c07252e9a5533b0, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:44.945837 27708 consensus_queue.cc:162] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:44.946123 27708 raft_consensus.cc:314] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:54:44.946281 27708 raft_consensus.cc:370] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:44.946411 27708 raft_consensus.cc:2019] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:44.947978 27708 raft_consensus.cc:1970] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.628s I0628 23:54:44.948220 27708 raft_consensus.cc:386] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "b06d8ce417fb408f8c07252e9a5533b0" member_type: VOTER } I0628 23:54:44.950350 27708 leader_election.cc:248] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:44.952013 27713 raft_consensus.cc:1970] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.587s I0628 23:54:44.952332 27713 raft_consensus.cc:1859] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:44.954645 27713 raft_consensus.cc:464] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 1 LEADER]: Becoming Leader. State: Replica: b06d8ce417fb408f8c07252e9a5533b0, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:44.955474 27713 consensus_queue.cc:145] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b06d8ce417fb408f8c07252e9a5533b0" member_type: VOTER } I0628 23:54:44.974272 27708 sys_catalog.cc:235] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "b06d8ce417fb408f8c07252e9a5533b0" config { opid_index: -1 peers { permanent_uuid: "b06d8ce417fb408f8c07252e9a5533b0" member_type: VOTER } } I0628 23:54:44.974690 27708 sys_catalog.cc:238] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [sys.catalog]: This master's current role is: LEADER I0628 23:54:44.976141 27715 sys_catalog.cc:235] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b06d8ce417fb408f8c07252e9a5533b0. Latest consensus state: current_term: 1 leader_uuid: "b06d8ce417fb408f8c07252e9a5533b0" config { opid_index: -1 peers { permanent_uuid: "b06d8ce417fb408f8c07252e9a5533b0" member_type: VOTER } } I0628 23:54:44.976785 27715 sys_catalog.cc:238] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [sys.catalog]: This master's current role is: LEADER I0628 23:54:44.976099 27714 sys_catalog.cc:235] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b06d8ce417fb408f8c07252e9a5533b0" config { opid_index: -1 peers { permanent_uuid: "b06d8ce417fb408f8c07252e9a5533b0" member_type: VOTER } } I0628 23:54:44.980568 27714 sys_catalog.cc:238] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [sys.catalog]: This master's current role is: LEADER I0628 23:54:44.981675 27708 sys_catalog.cc:311] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:54:44.983155 27665 master_main.cc:64] Master server successfully started. I0628 23:54:44.984041 27717 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:54:44.985314 27718 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P b06d8ce417fb408f8c07252e9a5533b0 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:54:45.023820 27709 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:45.024401 27709 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:45.038002 27709 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:45.040796 27709 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 128851 I0628 23:54:45.046720 27709 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:54:45.047134 27709 server_base.cc:165] Creating new FS layout I0628 23:54:45.055166 27709 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "fad53de7daf340bdbf96055a00e70a52" format_stamp: "Formatted at 2016-06-28 23:54:45 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:45.063449 27709 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-0 uuid: "fad53de7daf340bdbf96055a00e70a52" format_stamp: "Formatted at 2016-06-28 23:54:45 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:45.108536 27709 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:45.475262 27709 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:52855 I0628 23:54:45.477473 27709 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:54:45.477596 27709 webserver.cc:133] Document root disabled I0628 23:54:45.479166 27709 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:46435/ I0628 23:54:45.480654 27709 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:54:45.483280 27709 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:45.489672 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 27709 I0628 23:54:45.490731 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:39993 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:45.499011 27833 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:39993 I0628 23:54:45.499433 27833 heartbeater.cc:319] Registering TS with master... I0628 23:54:45.499696 27833 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:45.501428 27685 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "fad53de7daf340bdbf96055a00e70a52" instance_seqno: 1467158085453877 } with Master I0628 23:54:45.607674 27837 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:45.608136 27837 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:45.622234 27837 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:45.625047 27837 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 128851 I0628 23:54:45.631096 27837 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:54:45.631289 27837 server_base.cc:165] Creating new FS layout I0628 23:54:45.639395 27837 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "60ef03f1cf2e47f49194d736f48965bd" format_stamp: "Formatted at 2016-06-28 23:54:45 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:45.647253 27837 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-1 uuid: "60ef03f1cf2e47f49194d736f48965bd" format_stamp: "Formatted at 2016-06-28 23:54:45 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:45.680567 27837 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:45.773677 27837 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:44250 I0628 23:54:45.775123 27837 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:54:45.775219 27837 webserver.cc:133] Document root disabled I0628 23:54:45.776671 27837 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:36820/ I0628 23:54:45.778270 27837 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:54:45.780652 27837 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:45.788048 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 27837 I0628 23:54:45.788830 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:39993 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:45.789527 27951 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:39993 I0628 23:54:45.789868 27951 heartbeater.cc:319] Registering TS with master... I0628 23:54:45.790061 27951 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:45.791360 27685 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "60ef03f1cf2e47f49194d736f48965bd" instance_seqno: 1467158085755172 } with Master I0628 23:54:45.896284 27954 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:45.896733 27954 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:45.910089 27954 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:45.912834 27954 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 128851 I0628 23:54:45.918493 27954 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:54:45.918675 27954 server_base.cc:165] Creating new FS layout I0628 23:54:45.926298 27954 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "3a00359c210041bd86f5da4c403ed2af" format_stamp: "Formatted at 2016-06-28 23:54:45 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:45.934259 27954 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-2 uuid: "3a00359c210041bd86f5da4c403ed2af" format_stamp: "Formatted at 2016-06-28 23:54:45 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:45.968627 27954 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:46.058722 27954 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:37991 I0628 23:54:46.060714 27954 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:54:46.060822 27954 webserver.cc:133] Document root disabled I0628 23:54:46.062445 27954 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:51458/ I0628 23:54:46.063951 27954 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableDestUuidValidation.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:54:46.065095 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 27954 I0628 23:54:46.066529 27954 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:46.077755 28068 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:39993 I0628 23:54:46.078174 28068 heartbeater.cc:319] Registering TS with master... I0628 23:54:46.078454 28068 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:46.079687 27685 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "3a00359c210041bd86f5da4c403ed2af" instance_seqno: 1467158086040089 } with Master I0628 23:54:46.081732 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:54:46.129351 27685 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:43649: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:54:46.172883 27888 tablet_service.cc:612] Processing CreateTablet for tablet fa5f39316f3a4636b5ca8390b94b8183 (table=test-workload [id=53a1f725371e4fc08adc753b8cb4442f]), partition=range: [(), ()) I0628 23:54:46.174875 27770 tablet_service.cc:612] Processing CreateTablet for tablet fa5f39316f3a4636b5ca8390b94b8183 (table=test-workload [id=53a1f725371e4fc08adc753b8cb4442f]), partition=range: [(), ()) I0628 23:54:46.184384 28005 tablet_service.cc:612] Processing CreateTablet for tablet fa5f39316f3a4636b5ca8390b94b8183 (table=test-workload [id=53a1f725371e4fc08adc753b8cb4442f]), partition=range: [(), ()) I0628 23:54:46.189843 27888 ts_tablet_manager.cc:760] Registered tablet fa5f39316f3a4636b5ca8390b94b8183 I0628 23:54:46.191539 28087 ts_tablet_manager.cc:628] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd: Bootstrapping tablet I0628 23:54:46.194741 28087 tablet_bootstrap.cc:393] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd: Bootstrap starting. I0628 23:54:46.199780 28087 tablet_bootstrap.cc:553] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:46.200393 28087 tablet_bootstrap.cc:493] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd: No blocks or log segments found. Creating new log. I0628 23:54:46.202647 28087 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:46.213058 27770 ts_tablet_manager.cc:760] Registered tablet fa5f39316f3a4636b5ca8390b94b8183 I0628 23:54:46.214851 28005 ts_tablet_manager.cc:760] Registered tablet fa5f39316f3a4636b5ca8390b94b8183 I0628 23:54:46.219769 28091 ts_tablet_manager.cc:628] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af: Bootstrapping tablet I0628 23:54:46.221587 28087 tablet_bootstrap.cc:393] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd: No bootstrap required, opened a new log I0628 23:54:46.222285 28087 ts_tablet_manager.cc:633] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.016s I0628 23:54:46.222689 28090 ts_tablet_manager.cc:628] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52: Bootstrapping tablet I0628 23:54:46.223580 28091 tablet_bootstrap.cc:393] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af: Bootstrap starting. I0628 23:54:46.225100 28090 tablet_bootstrap.cc:393] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52: Bootstrap starting. I0628 23:54:46.228888 28091 tablet_bootstrap.cc:553] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:54:46.229610 28091 tablet_bootstrap.cc:493] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af: No blocks or log segments found. Creating new log. I0628 23:54:46.230140 28090 tablet_bootstrap.cc:553] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52: Time spent opening tablet: real 0.003s user 0.004s sys 0.000s I0628 23:54:46.230854 28090 tablet_bootstrap.cc:493] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52: No blocks or log segments found. Creating new log. I0628 23:54:46.232244 28091 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:46.231245 28087 raft_consensus.cc:269] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "60ef03f1cf2e47f49194d736f48965bd" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 44250 } } peers { permanent_uuid: "fad53de7daf340bdbf96055a00e70a52" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 52855 } } peers { permanent_uuid: "3a00359c210041bd86f5da4c403ed2af" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37991 } } I0628 23:54:46.232995 28090 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:46.233667 28087 raft_consensus.cc:301] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:46.234019 28087 raft_consensus.cc:499] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ef03f1cf2e47f49194d736f48965bd, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:46.234628 28087 consensus_queue.cc:162] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:46.237994 28090 tablet_bootstrap.cc:393] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52: No bootstrap required, opened a new log I0628 23:54:46.238373 28090 ts_tablet_manager.cc:633] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s I0628 23:54:46.246029 28087 ts_tablet_manager.cc:659] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd: Time spent starting tablet: real 0.023s user 0.016s sys 0.000s I0628 23:54:46.248615 28090 raft_consensus.cc:269] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "60ef03f1cf2e47f49194d736f48965bd" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 44250 } } peers { permanent_uuid: "fad53de7daf340bdbf96055a00e70a52" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 52855 } } peers { permanent_uuid: "3a00359c210041bd86f5da4c403ed2af" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37991 } } I0628 23:54:46.250401 28090 raft_consensus.cc:301] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:46.250705 28090 raft_consensus.cc:499] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad53de7daf340bdbf96055a00e70a52, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:46.251123 28090 consensus_queue.cc:162] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:46.256413 28090 ts_tablet_manager.cc:659] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s I0628 23:54:46.257264 28091 tablet_bootstrap.cc:393] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af: No bootstrap required, opened a new log I0628 23:54:46.257724 28091 ts_tablet_manager.cc:633] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.024s I0628 23:54:46.265264 28091 raft_consensus.cc:269] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "60ef03f1cf2e47f49194d736f48965bd" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 44250 } } peers { permanent_uuid: "fad53de7daf340bdbf96055a00e70a52" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 52855 } } peers { permanent_uuid: "3a00359c210041bd86f5da4c403ed2af" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37991 } } I0628 23:54:46.266647 28091 raft_consensus.cc:301] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:46.266890 28091 raft_consensus.cc:499] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a00359c210041bd86f5da4c403ed2af, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:46.267336 28091 consensus_queue.cc:162] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:46.270359 28091 ts_tablet_manager.cc:659] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s I0628 23:54:46.666218 28093 raft_consensus.cc:370] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:46.666442 28093 raft_consensus.cc:2019] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:46.668663 28093 raft_consensus.cc:1970] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.594s I0628 23:54:46.668864 28093 raft_consensus.cc:386] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "60ef03f1cf2e47f49194d736f48965bd" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 44250 } } peers { permanent_uuid: "fad53de7daf340bdbf96055a00e70a52" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 52855 } } peers { permanent_uuid: "3a00359c210041bd86f5da4c403ed2af" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37991 } } I0628 23:54:46.672171 28093 leader_election.cc:223] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [CANDIDATE]: Term 1 election: Requesting vote from peer fad53de7daf340bdbf96055a00e70a52 I0628 23:54:46.672709 28093 leader_election.cc:223] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [CANDIDATE]: Term 1 election: Requesting vote from peer 3a00359c210041bd86f5da4c403ed2af I0628 23:54:46.678725 27790 raft_consensus.cc:2019] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:46.681321 28025 raft_consensus.cc:2019] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:46.681818 27790 raft_consensus.cc:1970] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.617s I0628 23:54:46.684324 28025 raft_consensus.cc:1970] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.595s I0628 23:54:46.684458 27790 raft_consensus.cc:1677] T fa5f39316f3a4636b5ca8390b94b8183 P fad53de7daf340bdbf96055a00e70a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ef03f1cf2e47f49194d736f48965bd in term 1. I0628 23:54:46.685765 27842 leader_election.cc:347] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [CANDIDATE]: Term 1 election: Vote granted by peer fad53de7daf340bdbf96055a00e70a52 I0628 23:54:46.686115 27842 leader_election.cc:248] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:46.686550 28025 raft_consensus.cc:1677] T fa5f39316f3a4636b5ca8390b94b8183 P 3a00359c210041bd86f5da4c403ed2af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ef03f1cf2e47f49194d736f48965bd in term 1. I0628 23:54:46.686894 28096 raft_consensus.cc:1970] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.643s I0628 23:54:46.687188 28096 raft_consensus.cc:1859] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:46.687497 27842 leader_election.cc:347] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [CANDIDATE]: Term 1 election: Vote granted by peer 3a00359c210041bd86f5da4c403ed2af I0628 23:54:46.688935 28096 raft_consensus.cc:464] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 1 LEADER]: Becoming Leader. State: Replica: 60ef03f1cf2e47f49194d736f48965bd, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:46.689923 28096 consensus_queue.cc:145] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "60ef03f1cf2e47f49194d736f48965bd" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 44250 } } peers { permanent_uuid: "fad53de7daf340bdbf96055a00e70a52" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 52855 } } peers { permanent_uuid: "3a00359c210041bd86f5da4c403ed2af" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37991 } } I0628 23:54:46.715431 27684 catalog_manager.cc:1673] Tablet: fa5f39316f3a4636b5ca8390b94b8183 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "60ef03f1cf2e47f49194d736f48965bd" config { opid_index: -1 peers { permanent_uuid: "60ef03f1cf2e47f49194d736f48965bd" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 44250 } } peers { permanent_uuid: "fad53de7daf340bdbf96055a00e70a52" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 52855 } } peers { permanent_uuid: "3a00359c210041bd86f5da4c403ed2af" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 37991 } } } W0628 23:54:46.724874 28109 raft_consensus_state.cc:541] T fa5f39316f3a4636b5ca8390b94b8183 P 60ef03f1cf2e47f49194d736f48965bd [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 W0628 23:54:46.763515 27770 tablet_service.cc:186] Invalid argument: DeleteTablet: Wrong destination UUID requested. Local UUID: fad53de7daf340bdbf96055a00e70a52. Requested UUID: fake-uuid: from {real_user=slave, eff_user=} at 127.0.0.1:44030: tablet_id: "fa5f39316f3a4636b5ca8390b94b8183" delete_type: TABLET_DATA_TOMBSTONED dest_uuid: "fake-uuid" I0628 23:54:46.776685 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 27665 I0628 23:54:46.793792 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 27709 I0628 23:54:46.808145 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 27837 I0628 23:54:46.822830 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 27954 [ OK ] DeleteTableTest.TestDeleteTableDestUuidValidation (2174 ms) [ RUN ] DeleteTableTest.TestAtomicDeleteTablet I0628 23:54:46.845762 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:46.953415 28112 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:46.954046 28112 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:46.965816 28112 master_main.cc:58] Initializing master server... I0628 23:54:46.967955 28112 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 129351 I0628 23:54:46.974216 28112 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:54:46.974412 28112 server_base.cc:165] Creating new FS layout I0628 23:54:46.987270 28112 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/master/instance: uuid: "1ad263b475ad4ea38c9d39c30907abcb" format_stamp: "Formatted at 2016-06-28 23:54:46 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:46.999713 28112 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/master uuid: "1ad263b475ad4ea38c9d39c30907abcb" format_stamp: "Formatted at 2016-06-28 23:54:46 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:47.030423 28112 master_main.cc:61] Starting Master server... I0628 23:54:47.063930 28112 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:34905 I0628 23:54:47.065335 28112 webserver.cc:122] Starting webserver on localhost:0 I0628 23:54:47.065434 28112 webserver.cc:133] Document root disabled I0628 23:54:47.066154 28112 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:54:47.067431 28112 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:50998/ I0628 23:54:47.068681 28112 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:54:47.071315 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 28112 I0628 23:54:47.072348 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:34905 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:47.094969 28155 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb: Bootstrap starting. I0628 23:54:47.098515 28155 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:47.099175 28155 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb: No blocks or log segments found. Creating new log. I0628 23:54:47.100946 28155 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:47.105341 28155 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb: No bootstrap required, opened a new log I0628 23:54:47.111212 28155 raft_consensus.cc:269] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "1ad263b475ad4ea38c9d39c30907abcb" member_type: VOTER } I0628 23:54:47.112222 28155 raft_consensus.cc:301] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:47.112520 28155 raft_consensus.cc:499] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad263b475ad4ea38c9d39c30907abcb, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:47.112928 28155 consensus_queue.cc:162] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:47.113211 28155 raft_consensus.cc:314] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:54:47.113379 28155 raft_consensus.cc:370] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:47.113514 28155 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:47.115017 28155 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.501s I0628 23:54:47.115260 28155 raft_consensus.cc:386] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "1ad263b475ad4ea38c9d39c30907abcb" member_type: VOTER } I0628 23:54:47.117518 28155 leader_election.cc:248] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:47.119129 28160 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.558s I0628 23:54:47.119436 28160 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:47.121829 28160 raft_consensus.cc:464] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 1 LEADER]: Becoming Leader. State: Replica: 1ad263b475ad4ea38c9d39c30907abcb, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:47.122469 28160 consensus_queue.cc:145] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "1ad263b475ad4ea38c9d39c30907abcb" member_type: VOTER } I0628 23:54:47.140797 28155 sys_catalog.cc:235] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "1ad263b475ad4ea38c9d39c30907abcb" config { opid_index: -1 peers { permanent_uuid: "1ad263b475ad4ea38c9d39c30907abcb" member_type: VOTER } } I0628 23:54:47.141219 28155 sys_catalog.cc:238] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [sys.catalog]: This master's current role is: LEADER I0628 23:54:47.143661 28164 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:54:47.142590 28161 sys_catalog.cc:235] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ad263b475ad4ea38c9d39c30907abcb" config { opid_index: -1 peers { permanent_uuid: "1ad263b475ad4ea38c9d39c30907abcb" member_type: VOTER } } I0628 23:54:47.147219 28155 sys_catalog.cc:311] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [sys.catalog]: configured and running, proceeding with master startup. I0628 23:54:47.147182 28162 sys_catalog.cc:235] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ad263b475ad4ea38c9d39c30907abcb. Latest consensus state: current_term: 1 leader_uuid: "1ad263b475ad4ea38c9d39c30907abcb" config { opid_index: -1 peers { permanent_uuid: "1ad263b475ad4ea38c9d39c30907abcb" member_type: VOTER } } I0628 23:54:47.148108 28162 sys_catalog.cc:238] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [sys.catalog]: This master's current role is: LEADER I0628 23:54:47.148037 28161 sys_catalog.cc:238] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [sys.catalog]: This master's current role is: LEADER I0628 23:54:47.150809 28165 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 1ad263b475ad4ea38c9d39c30907abcb [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:54:47.157829 28112 master_main.cc:64] Master server successfully started. I0628 23:54:47.189060 28156 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:47.189563 28156 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:47.213006 28156 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:47.216158 28156 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 129851 I0628 23:54:47.223158 28156 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:54:47.223367 28156 server_base.cc:165] Creating new FS layout I0628 23:54:47.232213 28156 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "ebac69556c834c3f9c74fea0200c2193" format_stamp: "Formatted at 2016-06-28 23:54:47 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:47.240901 28156 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-0 uuid: "ebac69556c834c3f9c74fea0200c2193" format_stamp: "Formatted at 2016-06-28 23:54:47 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:47.276633 28156 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:47.754279 28156 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:34927 I0628 23:54:47.755646 28156 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:54:47.755754 28156 webserver.cc:133] Document root disabled I0628 23:54:47.757333 28156 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:34884/ I0628 23:54:47.758594 28156 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:54:47.760846 28156 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:47.764701 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 28156 I0628 23:54:47.765720 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:34905 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:47.774391 28280 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:34905 I0628 23:54:47.774760 28280 heartbeater.cc:319] Registering TS with master... I0628 23:54:47.774968 28280 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:47.776634 28132 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "ebac69556c834c3f9c74fea0200c2193" instance_seqno: 1467158087616580 } with Master I0628 23:54:47.876219 28284 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:47.876658 28284 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:47.889930 28284 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:47.892643 28284 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 129851 I0628 23:54:47.898563 28284 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:54:47.898744 28284 server_base.cc:165] Creating new FS layout I0628 23:54:47.906391 28284 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "06111be5a9124ccba7d1c657d1348e8d" format_stamp: "Formatted at 2016-06-28 23:54:47 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:47.914312 28284 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-1 uuid: "06111be5a9124ccba7d1c657d1348e8d" format_stamp: "Formatted at 2016-06-28 23:54:47 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:47.948084 28284 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:48.070507 28284 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:37846 I0628 23:54:48.072407 28284 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:54:48.072546 28284 webserver.cc:133] Document root disabled I0628 23:54:48.074506 28284 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:55143/ I0628 23:54:48.075757 28284 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:54:48.082962 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 28284 I0628 23:54:48.083891 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:34905 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:48.084169 28284 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:48.091707 28398 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:34905 I0628 23:54:48.092083 28398 heartbeater.cc:319] Registering TS with master... I0628 23:54:48.092334 28398 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:48.093555 28132 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "06111be5a9124ccba7d1c657d1348e8d" instance_seqno: 1467158088023299 } with Master I0628 23:54:48.205535 28401 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:48.206259 28401 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:48.220016 28401 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:48.223541 28401 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 130351 I0628 23:54:48.231921 28401 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:54:48.232188 28401 server_base.cc:165] Creating new FS layout I0628 23:54:48.240687 28401 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" format_stamp: "Formatted at 2016-06-28 23:54:48 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:48.248441 28401 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-2 uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" format_stamp: "Formatted at 2016-06-28 23:54:48 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:48.282505 28401 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:48.379510 28401 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:38968 I0628 23:54:48.381067 28401 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:54:48.381173 28401 webserver.cc:133] Document root disabled I0628 23:54:48.382661 28401 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:43830/ I0628 23:54:48.384004 28401 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAtomicDeleteTablet.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:54:48.386596 28401 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:48.391264 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 28401 I0628 23:54:48.397498 28515 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:34905 I0628 23:54:48.397775 28515 heartbeater.cc:319] Registering TS with master... I0628 23:54:48.397964 28515 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:48.399245 28132 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" instance_seqno: 1467158088360649 } with Master I0628 23:54:48.401855 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:54:48.426717 28132 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:34797: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:54:48.470780 28335 tablet_service.cc:612] Processing CreateTablet for tablet ac874b2c8f7d48ce9e41330eb236a217 (table=test-workload [id=e5aedb8559934261870dabd85a7c390c]), partition=range: [(), ()) I0628 23:54:48.489398 28335 ts_tablet_manager.cc:760] Registered tablet ac874b2c8f7d48ce9e41330eb236a217 I0628 23:54:48.490036 28217 tablet_service.cc:612] Processing CreateTablet for tablet ac874b2c8f7d48ce9e41330eb236a217 (table=test-workload [id=e5aedb8559934261870dabd85a7c390c]), partition=range: [(), ()) I0628 23:54:48.493757 28452 tablet_service.cc:612] Processing CreateTablet for tablet ac874b2c8f7d48ce9e41330eb236a217 (table=test-workload [id=e5aedb8559934261870dabd85a7c390c]), partition=range: [(), ()) I0628 23:54:48.509032 28535 ts_tablet_manager.cc:628] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d: Bootstrapping tablet I0628 23:54:48.512269 28535 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d: Bootstrap starting. I0628 23:54:48.517305 28452 ts_tablet_manager.cc:760] Registered tablet ac874b2c8f7d48ce9e41330eb236a217 I0628 23:54:48.517727 28535 tablet_bootstrap.cc:553] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:48.518412 28535 tablet_bootstrap.cc:493] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d: No blocks or log segments found. Creating new log. I0628 23:54:48.519538 28217 ts_tablet_manager.cc:760] Registered tablet ac874b2c8f7d48ce9e41330eb236a217 I0628 23:54:48.520776 28535 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:48.524621 28537 ts_tablet_manager.cc:628] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe: Bootstrapping tablet I0628 23:54:48.526612 28538 ts_tablet_manager.cc:628] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Bootstrapping tablet I0628 23:54:48.528825 28537 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe: Bootstrap starting. I0628 23:54:48.529959 28538 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Bootstrap starting. I0628 23:54:48.534374 28537 tablet_bootstrap.cc:553] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:54:48.535001 28537 tablet_bootstrap.cc:493] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe: No blocks or log segments found. Creating new log. I0628 23:54:48.535619 28538 tablet_bootstrap.cc:553] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Time spent opening tablet: real 0.002s user 0.000s sys 0.004s I0628 23:54:48.536201 28538 tablet_bootstrap.cc:493] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: No blocks or log segments found. Creating new log. I0628 23:54:48.537438 28537 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:48.538640 28538 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:48.538961 28535 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d: No bootstrap required, opened a new log I0628 23:54:48.539317 28535 ts_tablet_manager.cc:633] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.000s I0628 23:54:48.543619 28538 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: No bootstrap required, opened a new log I0628 23:54:48.544080 28538 ts_tablet_manager.cc:633] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s I0628 23:54:48.550386 28537 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe: No bootstrap required, opened a new log I0628 23:54:48.550756 28537 ts_tablet_manager.cc:633] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.008s I0628 23:54:48.551173 28538 raft_consensus.cc:269] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "06111be5a9124ccba7d1c657d1348e8d" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 37846 } } peers { permanent_uuid: "ebac69556c834c3f9c74fea0200c2193" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34927 } } peers { permanent_uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 38968 } } I0628 23:54:48.552866 28538 raft_consensus.cc:301] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:48.553220 28538 raft_consensus.cc:499] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebac69556c834c3f9c74fea0200c2193, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:48.553764 28538 consensus_queue.cc:162] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:48.556654 28538 ts_tablet_manager.cc:659] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s I0628 23:54:48.558533 28537 raft_consensus.cc:269] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "06111be5a9124ccba7d1c657d1348e8d" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 37846 } } peers { permanent_uuid: "ebac69556c834c3f9c74fea0200c2193" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34927 } } peers { permanent_uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 38968 } } I0628 23:54:48.560603 28537 raft_consensus.cc:301] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:48.559957 28535 raft_consensus.cc:269] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "06111be5a9124ccba7d1c657d1348e8d" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 37846 } } peers { permanent_uuid: "ebac69556c834c3f9c74fea0200c2193" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34927 } } peers { permanent_uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 38968 } } I0628 23:54:48.560992 28537 raft_consensus.cc:499] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea3c3213caf4bf6b4d4b535716f5bfe, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:48.561594 28537 consensus_queue.cc:162] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:48.562134 28535 raft_consensus.cc:301] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:48.562439 28535 raft_consensus.cc:499] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06111be5a9124ccba7d1c657d1348e8d, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:48.562980 28535 consensus_queue.cc:162] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:48.566454 28537 ts_tablet_manager.cc:659] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe: Time spent starting tablet: real 0.016s user 0.008s sys 0.004s I0628 23:54:48.567248 28535 ts_tablet_manager.cc:659] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d: Time spent starting tablet: real 0.028s user 0.012s sys 0.016s I0628 23:54:48.832445 28548 raft_consensus.cc:370] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:48.832625 28548 raft_consensus.cc:2019] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:48.834635 28548 raft_consensus.cc:1970] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.513s I0628 23:54:48.834940 28548 raft_consensus.cc:386] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "06111be5a9124ccba7d1c657d1348e8d" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 37846 } } peers { permanent_uuid: "ebac69556c834c3f9c74fea0200c2193" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34927 } } peers { permanent_uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 38968 } } I0628 23:54:48.838903 28548 leader_election.cc:223] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [CANDIDATE]: Term 1 election: Requesting vote from peer 06111be5a9124ccba7d1c657d1348e8d I0628 23:54:48.839381 28548 leader_election.cc:223] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [CANDIDATE]: Term 1 election: Requesting vote from peer ebac69556c834c3f9c74fea0200c2193 I0628 23:54:48.845525 28355 raft_consensus.cc:2019] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:48.845686 28237 raft_consensus.cc:2019] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:48.847348 28355 raft_consensus.cc:1970] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.601s I0628 23:54:48.847579 28237 raft_consensus.cc:1970] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.625s I0628 23:54:48.849445 28355 raft_consensus.cc:1677] T ac874b2c8f7d48ce9e41330eb236a217 P 06111be5a9124ccba7d1c657d1348e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea3c3213caf4bf6b4d4b535716f5bfe in term 1. I0628 23:54:48.849745 28237 raft_consensus.cc:1677] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea3c3213caf4bf6b4d4b535716f5bfe in term 1. I0628 23:54:48.850730 28408 leader_election.cc:347] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [CANDIDATE]: Term 1 election: Vote granted by peer 06111be5a9124ccba7d1c657d1348e8d I0628 23:54:48.851192 28406 leader_election.cc:347] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [CANDIDATE]: Term 1 election: Vote granted by peer ebac69556c834c3f9c74fea0200c2193 I0628 23:54:48.851390 28406 leader_election.cc:248] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:48.851984 28549 raft_consensus.cc:1970] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.590s I0628 23:54:48.852242 28549 raft_consensus.cc:1859] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:48.853703 28549 raft_consensus.cc:464] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 1 LEADER]: Becoming Leader. State: Replica: 4ea3c3213caf4bf6b4d4b535716f5bfe, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:48.854475 28549 consensus_queue.cc:145] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "06111be5a9124ccba7d1c657d1348e8d" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 37846 } } peers { permanent_uuid: "ebac69556c834c3f9c74fea0200c2193" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34927 } } peers { permanent_uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 38968 } } W0628 23:54:48.880376 28555 raft_consensus_state.cc:541] T ac874b2c8f7d48ce9e41330eb236a217 P 4ea3c3213caf4bf6b4d4b535716f5bfe [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:54:48.879899 28130 catalog_manager.cc:1673] Tablet: ac874b2c8f7d48ce9e41330eb236a217 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" config { opid_index: -1 peers { permanent_uuid: "06111be5a9124ccba7d1c657d1348e8d" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 37846 } } peers { permanent_uuid: "ebac69556c834c3f9c74fea0200c2193" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34927 } } peers { permanent_uuid: "4ea3c3213caf4bf6b4d4b535716f5bfe" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 38968 } } } I0628 23:54:48.936681 28217 tablet_service.cc:653] Processing DeleteTablet for tablet ac874b2c8f7d48ce9e41330eb236a217 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:51567 I0628 23:54:48.938894 28217 tablet_service.cc:653] Processing DeleteTablet for tablet ac874b2c8f7d48ce9e41330eb236a217 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:51567 I0628 23:54:48.939302 28217 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: ac874b2c8f7d48ce9e41330eb236a217 I0628 23:54:48.939658 28217 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(ac874b2c8f7d48ce9e41330eb236a217) I0628 23:54:48.939867 28217 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(ac874b2c8f7d48ce9e41330eb236a217) I0628 23:54:48.940109 28217 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(ac874b2c8f7d48ce9e41330eb236a217) I0628 23:54:48.940352 28217 maintenance_manager.cc:180] Unregistered op FlushMRSOp(ac874b2c8f7d48ce9e41330eb236a217) I0628 23:54:48.940510 28217 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(ac874b2c8f7d48ce9e41330eb236a217) I0628 23:54:48.940665 28217 maintenance_manager.cc:180] Unregistered op LogGCOp(ac874b2c8f7d48ce9e41330eb236a217) I0628 23:54:48.940820 28217 raft_consensus.cc:1495] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:54:48.941150 28217 raft_consensus.cc:1511] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:54:48.945468 28217 ts_tablet_manager.cc:955] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:54:48.951385 28217 ts_tablet_manager.cc:965] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Tablet deleted. Last logged OpId: 0.0 I0628 23:54:48.952330 28217 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Deleted tablet blocks from disk I0628 23:54:48.958454 28217 tablet_service.cc:653] Processing DeleteTablet for tablet ac874b2c8f7d48ce9e41330eb236a217 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:51567 I0628 23:54:48.958755 28217 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: ac874b2c8f7d48ce9e41330eb236a217 I0628 23:54:48.958902 28217 ts_tablet_manager.cc:955] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:54:48.964426 28217 ts_tablet_manager.cc:965] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Tablet deleted. Last logged OpId: 0.0 I0628 23:54:48.964747 28217 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Deleted tablet blocks from disk I0628 23:54:48.968431 28217 tablet_service.cc:653] Processing DeleteTablet for tablet ac874b2c8f7d48ce9e41330eb236a217 with delete_type TABLET_DATA_DELETED from {real_user=slave, eff_user=} at 127.0.0.1:51567 I0628 23:54:48.968668 28217 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: ac874b2c8f7d48ce9e41330eb236a217 I0628 23:54:48.968794 28217 ts_tablet_manager.cc:955] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:54:48.972359 28217 ts_tablet_manager.cc:965] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Tablet deleted. Last logged OpId: 0.0 I0628 23:54:48.972960 28217 tablet_bootstrap.cc:393] T ac874b2c8f7d48ce9e41330eb236a217 P ebac69556c834c3f9c74fea0200c2193: Deleted tablet blocks from disk I0628 23:54:48.975178 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 28112 I0628 23:54:48.992142 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 28156 I0628 23:54:49.007702 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 28284 I0628 23:54:49.022162 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 28401 [ OK ] DeleteTableTest.TestAtomicDeleteTablet (2199 ms) [ RUN ] DeleteTableTest.TestDeleteTableWithConcurrentWrites I0628 23:54:49.044515 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:49.156417 28558 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:49.156903 28558 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:49.168470 28558 master_main.cc:58] Initializing master server... I0628 23:54:49.170929 28558 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 130851 I0628 23:54:49.177326 28558 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:54:49.177522 28558 server_base.cc:165] Creating new FS layout I0628 23:54:49.191422 28558 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/master/instance: uuid: "68bd46bfcdfd456081e4d63b38403b38" format_stamp: "Formatted at 2016-06-28 23:54:49 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:49.203770 28558 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/master uuid: "68bd46bfcdfd456081e4d63b38403b38" format_stamp: "Formatted at 2016-06-28 23:54:49 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:49.236457 28558 master_main.cc:61] Starting Master server... I0628 23:54:49.271247 28558 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:32880 I0628 23:54:49.272637 28558 webserver.cc:122] Starting webserver on localhost:0 I0628 23:54:49.272749 28558 webserver.cc:133] Document root disabled I0628 23:54:49.273476 28558 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:54:49.274688 28558 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:48006/ I0628 23:54:49.275930 28558 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:54:49.280205 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 28558 I0628 23:54:49.281234 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:32880 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:49.303683 28601 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38: Bootstrap starting. I0628 23:54:49.307510 28601 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:49.308303 28601 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38: No blocks or log segments found. Creating new log. I0628 23:54:49.309933 28601 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:49.336599 28601 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38: No bootstrap required, opened a new log I0628 23:54:49.352771 28601 raft_consensus.cc:269] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "68bd46bfcdfd456081e4d63b38403b38" member_type: VOTER } I0628 23:54:49.354593 28601 raft_consensus.cc:301] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:49.354857 28601 raft_consensus.cc:499] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bd46bfcdfd456081e4d63b38403b38, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:49.355268 28601 consensus_queue.cc:162] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:49.355588 28601 raft_consensus.cc:314] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:54:49.355768 28601 raft_consensus.cc:370] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:49.355954 28601 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:49.358362 28601 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I0628 23:54:49.358646 28601 raft_consensus.cc:386] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "68bd46bfcdfd456081e4d63b38403b38" member_type: VOTER } I0628 23:54:49.360954 28601 leader_election.cc:248] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:49.372895 28607 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.630s I0628 23:54:49.373167 28607 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:49.385872 28607 raft_consensus.cc:464] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 1 LEADER]: Becoming Leader. State: Replica: 68bd46bfcdfd456081e4d63b38403b38, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:49.386520 28607 consensus_queue.cc:145] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "68bd46bfcdfd456081e4d63b38403b38" member_type: VOTER } I0628 23:54:49.397065 28602 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:49.397603 28602 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:49.405946 28608 sys_catalog.cc:235] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bd46bfcdfd456081e4d63b38403b38. Latest consensus state: current_term: 1 leader_uuid: "68bd46bfcdfd456081e4d63b38403b38" config { opid_index: -1 peers { permanent_uuid: "68bd46bfcdfd456081e4d63b38403b38" member_type: VOTER } } I0628 23:54:49.406628 28608 sys_catalog.cc:238] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [sys.catalog]: This master's current role is: LEADER I0628 23:54:49.407614 28601 sys_catalog.cc:235] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "68bd46bfcdfd456081e4d63b38403b38" config { opid_index: -1 peers { permanent_uuid: "68bd46bfcdfd456081e4d63b38403b38" member_type: VOTER } } I0628 23:54:49.408601 28601 sys_catalog.cc:238] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [sys.catalog]: This master's current role is: LEADER I0628 23:54:49.411639 28610 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:54:49.408115 28609 sys_catalog.cc:235] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68bd46bfcdfd456081e4d63b38403b38" config { opid_index: -1 peers { permanent_uuid: "68bd46bfcdfd456081e4d63b38403b38" member_type: VOTER } } I0628 23:54:49.412559 28609 sys_catalog.cc:238] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [sys.catalog]: This master's current role is: LEADER I0628 23:54:49.414515 28602 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:49.415431 28601 sys_catalog.cc:311] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:54:49.416849 28558 master_main.cc:64] Master server successfully started. I0628 23:54:49.417527 28611 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:54:49.418057 28602 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 130851 I0628 23:54:49.427637 28602 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:54:49.427930 28602 server_base.cc:165] Creating new FS layout I0628 23:54:49.438580 28602 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "420c8e80c83b4919bd67708634ad5bd9" format_stamp: "Formatted at 2016-06-28 23:54:49 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:49.450466 28602 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-0 uuid: "420c8e80c83b4919bd67708634ad5bd9" format_stamp: "Formatted at 2016-06-28 23:54:49 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:49.512933 28602 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:49.961078 28602 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:44123 I0628 23:54:49.962460 28602 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:54:49.962609 28602 webserver.cc:133] Document root disabled I0628 23:54:49.964285 28602 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:46976/ I0628 23:54:49.965750 28602 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:54:49.972951 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 28602 I0628 23:54:49.974056 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:32880 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:49.978006 28602 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:49.999701 28726 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:32880 I0628 23:54:50.000066 28726 heartbeater.cc:319] Registering TS with master... I0628 23:54:50.000308 28726 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:50.001935 28578 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "420c8e80c83b4919bd67708634ad5bd9" instance_seqno: 1467158089937879 } with Master I0628 23:54:50.087352 28730 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:50.087787 28730 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:50.100854 28730 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:50.103431 28730 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 131351 I0628 23:54:50.109195 28730 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:54:50.109396 28730 server_base.cc:165] Creating new FS layout I0628 23:54:50.116902 28730 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "a19aaebe5ce646349f024b13675e09ed" format_stamp: "Formatted at 2016-06-28 23:54:50 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:50.124236 28730 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-1 uuid: "a19aaebe5ce646349f024b13675e09ed" format_stamp: "Formatted at 2016-06-28 23:54:50 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:50.156575 28730 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:50.270926 28730 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:38140 I0628 23:54:50.272366 28730 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:54:50.272469 28730 webserver.cc:133] Document root disabled I0628 23:54:50.274111 28730 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:47496/ I0628 23:54:50.275720 28730 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:54:50.282776 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 28730 I0628 23:54:50.283648 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:32880 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:50.288396 28730 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:50.295183 28844 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:32880 I0628 23:54:50.295627 28844 heartbeater.cc:319] Registering TS with master... I0628 23:54:50.295853 28844 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:50.297549 28578 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "a19aaebe5ce646349f024b13675e09ed" instance_seqno: 1467158090232918 } with Master I0628 23:54:50.401361 28847 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:50.401829 28847 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:50.415211 28847 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:50.418164 28847 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 131351 I0628 23:54:50.424031 28847 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:54:50.424234 28847 server_base.cc:165] Creating new FS layout I0628 23:54:50.432795 28847 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "7333285db9804dde9478033cabd5d781" format_stamp: "Formatted at 2016-06-28 23:54:50 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:50.440790 28847 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-2 uuid: "7333285db9804dde9478033cabd5d781" format_stamp: "Formatted at 2016-06-28 23:54:50 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:50.474212 28847 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:50.571305 28847 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:40485 I0628 23:54:50.572700 28847 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:54:50.572789 28847 webserver.cc:133] Document root disabled I0628 23:54:50.574216 28847 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:41545/ I0628 23:54:50.575582 28847 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:54:50.577787 28847 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:50.581324 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 28847 I0628 23:54:50.586136 28961 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:32880 I0628 23:54:50.586477 28961 heartbeater.cc:319] Registering TS with master... I0628 23:54:50.586683 28961 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:50.588322 28578 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "7333285db9804dde9478033cabd5d781" instance_seqno: 1467158090552617 } with Master I0628 23:54:50.591871 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:54:50.614846 28578 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:47355: name: "table-0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:54:50.655107 28781 tablet_service.cc:612] Processing CreateTablet for tablet a16b944a988b4388b55f8396b18aa8b9 (table=table-0 [id=8e56fdabbd3145cb936ad1912202dc42]), partition=range: [(), ()) I0628 23:54:50.656972 28898 tablet_service.cc:612] Processing CreateTablet for tablet a16b944a988b4388b55f8396b18aa8b9 (table=table-0 [id=8e56fdabbd3145cb936ad1912202dc42]), partition=range: [(), ()) I0628 23:54:50.659173 28663 tablet_service.cc:612] Processing CreateTablet for tablet a16b944a988b4388b55f8396b18aa8b9 (table=table-0 [id=8e56fdabbd3145cb936ad1912202dc42]), partition=range: [(), ()) I0628 23:54:50.670027 28781 ts_tablet_manager.cc:760] Registered tablet a16b944a988b4388b55f8396b18aa8b9 I0628 23:54:50.671480 28981 ts_tablet_manager.cc:628] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: Bootstrapping tablet I0628 23:54:50.673738 28663 ts_tablet_manager.cc:760] Registered tablet a16b944a988b4388b55f8396b18aa8b9 I0628 23:54:50.675185 28981 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: Bootstrap starting. I0628 23:54:50.675310 28983 ts_tablet_manager.cc:628] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: Bootstrapping tablet I0628 23:54:50.677435 28983 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: Bootstrap starting. I0628 23:54:50.678549 28981 tablet_bootstrap.cc:553] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:50.679008 28898 ts_tablet_manager.cc:760] Registered tablet a16b944a988b4388b55f8396b18aa8b9 I0628 23:54:50.679034 28981 tablet_bootstrap.cc:493] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: No blocks or log segments found. Creating new log. I0628 23:54:50.680749 28983 tablet_bootstrap.cc:553] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:54:50.680778 28981 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:50.681195 28983 tablet_bootstrap.cc:493] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: No blocks or log segments found. Creating new log. I0628 23:54:50.682734 28983 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:50.685776 28984 ts_tablet_manager.cc:628] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: Bootstrapping tablet I0628 23:54:50.686986 28983 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: No bootstrap required, opened a new log I0628 23:54:50.687283 28983 ts_tablet_manager.cc:633] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s I0628 23:54:50.687930 28984 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: Bootstrap starting. I0628 23:54:50.691344 28984 tablet_bootstrap.cc:553] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:50.691812 28984 tablet_bootstrap.cc:493] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: No blocks or log segments found. Creating new log. I0628 23:54:50.692860 28981 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: No bootstrap required, opened a new log I0628 23:54:50.693253 28981 ts_tablet_manager.cc:633] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.008s I0628 23:54:50.693428 28984 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:50.694368 28983 raft_consensus.cc:269] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "7333285db9804dde9478033cabd5d781" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 40485 } } peers { permanent_uuid: "420c8e80c83b4919bd67708634ad5bd9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 44123 } } peers { permanent_uuid: "a19aaebe5ce646349f024b13675e09ed" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38140 } } I0628 23:54:50.695906 28983 raft_consensus.cc:301] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:50.696183 28983 raft_consensus.cc:499] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420c8e80c83b4919bd67708634ad5bd9, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:50.696585 28983 consensus_queue.cc:162] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:50.697211 28984 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: No bootstrap required, opened a new log I0628 23:54:50.697537 28984 ts_tablet_manager.cc:633] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s I0628 23:54:50.700239 28983 ts_tablet_manager.cc:659] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0628 23:54:50.701226 28981 raft_consensus.cc:269] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "7333285db9804dde9478033cabd5d781" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 40485 } } peers { permanent_uuid: "420c8e80c83b4919bd67708634ad5bd9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 44123 } } peers { permanent_uuid: "a19aaebe5ce646349f024b13675e09ed" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38140 } } I0628 23:54:50.702601 28981 raft_consensus.cc:301] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:50.702886 28981 raft_consensus.cc:499] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19aaebe5ce646349f024b13675e09ed, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:50.703462 28981 consensus_queue.cc:162] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:50.705996 28984 raft_consensus.cc:269] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "7333285db9804dde9478033cabd5d781" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 40485 } } peers { permanent_uuid: "420c8e80c83b4919bd67708634ad5bd9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 44123 } } peers { permanent_uuid: "a19aaebe5ce646349f024b13675e09ed" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38140 } } I0628 23:54:50.707515 28984 raft_consensus.cc:301] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:50.707751 28984 raft_consensus.cc:499] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7333285db9804dde9478033cabd5d781, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:50.708189 28984 consensus_queue.cc:162] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:50.711976 28984 ts_tablet_manager.cc:659] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s I0628 23:54:50.714756 28981 ts_tablet_manager.cc:659] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: Time spent starting tablet: real 0.021s user 0.008s sys 0.012s I0628 23:54:50.902868 28994 raft_consensus.cc:370] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:50.903054 28994 raft_consensus.cc:2019] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:50.905051 28994 raft_consensus.cc:1970] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.588s I0628 23:54:50.905258 28994 raft_consensus.cc:386] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "7333285db9804dde9478033cabd5d781" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 40485 } } peers { permanent_uuid: "420c8e80c83b4919bd67708634ad5bd9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 44123 } } peers { permanent_uuid: "a19aaebe5ce646349f024b13675e09ed" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38140 } } I0628 23:54:50.908365 28994 leader_election.cc:223] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [CANDIDATE]: Term 1 election: Requesting vote from peer 420c8e80c83b4919bd67708634ad5bd9 I0628 23:54:50.908802 28994 leader_election.cc:223] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [CANDIDATE]: Term 1 election: Requesting vote from peer a19aaebe5ce646349f024b13675e09ed I0628 23:54:50.913939 28801 raft_consensus.cc:2019] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:50.913971 28683 raft_consensus.cc:2019] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:50.916000 28683 raft_consensus.cc:1970] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.537s I0628 23:54:50.916164 28801 raft_consensus.cc:1970] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.643s I0628 23:54:50.917613 28683 raft_consensus.cc:1677] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7333285db9804dde9478033cabd5d781 in term 1. I0628 23:54:50.917713 28801 raft_consensus.cc:1677] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7333285db9804dde9478033cabd5d781 in term 1. I0628 23:54:50.918573 28854 leader_election.cc:347] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [CANDIDATE]: Term 1 election: Vote granted by peer a19aaebe5ce646349f024b13675e09ed I0628 23:54:50.918781 28854 leader_election.cc:248] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:50.918990 28855 leader_election.cc:347] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [CANDIDATE]: Term 1 election: Vote granted by peer 420c8e80c83b4919bd67708634ad5bd9 I0628 23:54:50.919312 28996 raft_consensus.cc:1970] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.548s I0628 23:54:50.919565 28996 raft_consensus.cc:1859] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:50.920858 28996 raft_consensus.cc:464] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 LEADER]: Becoming Leader. State: Replica: 7333285db9804dde9478033cabd5d781, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:50.921531 28996 consensus_queue.cc:145] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7333285db9804dde9478033cabd5d781" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 40485 } } peers { permanent_uuid: "420c8e80c83b4919bd67708634ad5bd9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 44123 } } peers { permanent_uuid: "a19aaebe5ce646349f024b13675e09ed" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38140 } } W0628 23:54:50.945489 29001 raft_consensus_state.cc:541] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:54:50.945193 28576 catalog_manager.cc:1673] Tablet: a16b944a988b4388b55f8396b18aa8b9 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "7333285db9804dde9478033cabd5d781" config { opid_index: -1 peers { permanent_uuid: "7333285db9804dde9478033cabd5d781" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 40485 } } peers { permanent_uuid: "420c8e80c83b4919bd67708634ad5bd9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 44123 } } peers { permanent_uuid: "a19aaebe5ce646349f024b13675e09ed" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38140 } } } I0628 23:54:51.042121 28801 raft_consensus.cc:869] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 1 FOLLOWER]: Refusing update from remote peer 7333285db9804dde9478033cabd5d781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:54:51.042798 28683 raft_consensus.cc:869] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Refusing update from remote peer 7333285db9804dde9478033cabd5d781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:54:51.043894 28998 consensus_queue.cc:578] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [LEADER]: Connected to new peer: Peer: a19aaebe5ce646349f024b13675e09ed, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:54:51.044270 28996 consensus_queue.cc:578] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [LEADER]: Connected to new peer: Peer: 420c8e80c83b4919bd67708634ad5bd9, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:54:51.091902 29001 raft_consensus_state.cc:534] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 2 I0628 23:54:51.119041 28574 catalog_manager.cc:1036] Servicing DeleteTable request from {real_user=slave, eff_user=} at 127.0.0.1:47354: table { table_name: "table-0" } I0628 23:54:51.125597 28578 catalog_manager.cc:1542] Got report from deleted tablet a16b944a988b4388b55f8396b18aa8b9 (table table-0 [id=8e56fdabbd3145cb936ad1912202dc42]) (Table deleted at 2016-06-28 23:54:51 UTC): Sending delete request for this tablet I0628 23:54:51.125880 28578 catalog_manager.cc:2547] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38: Deleting tablet a16b944a988b4388b55f8396b18aa8b9 on peer a19aaebe5ce646349f024b13675e09ed with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) I0628 23:54:51.127220 28574 catalog_manager.cc:2531] Sending DeleteTablet for 3 replicas of tablet a16b944a988b4388b55f8396b18aa8b9 I0628 23:54:51.127197 28781 tablet_service.cc:653] Processing DeleteTablet for tablet a16b944a988b4388b55f8396b18aa8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:33865 I0628 23:54:51.127410 28574 catalog_manager.cc:2547] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38: Deleting tablet a16b944a988b4388b55f8396b18aa8b9 on peer 7333285db9804dde9478033cabd5d781 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) I0628 23:54:51.127523 28781 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: a16b944a988b4388b55f8396b18aa8b9 I0628 23:54:51.127809 28781 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.128005 28781 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.128213 28781 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.128379 28781 maintenance_manager.cc:180] Unregistered op FlushMRSOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.128495 28781 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.128600 28781 maintenance_manager.cc:180] Unregistered op LogGCOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.128505 28898 tablet_service.cc:653] Processing DeleteTablet for tablet a16b944a988b4388b55f8396b18aa8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:54208 I0628 23:54:51.128787 28898 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: a16b944a988b4388b55f8396b18aa8b9 I0628 23:54:51.129060 28898 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.129243 28898 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.129422 28898 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.129601 28898 maintenance_manager.cc:180] Unregistered op FlushMRSOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.129740 28898 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.129840 28898 maintenance_manager.cc:180] Unregistered op LogGCOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.129977 28898 raft_consensus.cc:1495] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 LEADER]: Raft consensus shutting down. I0628 23:54:51.130623 28898 consensus_peers.cc:351] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 -> Peer a19aaebe5ce646349f024b13675e09ed (127.106.77.1:38140): Closing peer: a19aaebe5ce646349f024b13675e09ed I0628 23:54:51.130810 28574 catalog_manager.cc:2547] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38: Deleting tablet a16b944a988b4388b55f8396b18aa8b9 on peer 420c8e80c83b4919bd67708634ad5bd9 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) I0628 23:54:51.131139 28574 catalog_manager.cc:2547] T 00000000000000000000000000000000 P 68bd46bfcdfd456081e4d63b38403b38: Deleting tablet a16b944a988b4388b55f8396b18aa8b9 on peer a19aaebe5ce646349f024b13675e09ed with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) I0628 23:54:51.131896 28663 tablet_service.cc:653] Processing DeleteTablet for tablet a16b944a988b4388b55f8396b18aa8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:54991 I0628 23:54:51.131968 28574 catalog_manager.cc:1119] Successfully deleted table table-0 [id=8e56fdabbd3145cb936ad1912202dc42] per request from {real_user=slave, eff_user=} at 127.0.0.1:47354 I0628 23:54:51.132201 28663 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: a16b944a988b4388b55f8396b18aa8b9 I0628 23:54:51.132184 28780 tablet_service.cc:653] Processing DeleteTablet for tablet a16b944a988b4388b55f8396b18aa8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:33865 I0628 23:54:51.132462 28663 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.132624 28663 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.132757 28663 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.132917 28663 maintenance_manager.cc:180] Unregistered op FlushMRSOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.133044 28663 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.133168 28663 maintenance_manager.cc:180] Unregistered op LogGCOp(a16b944a988b4388b55f8396b18aa8b9) I0628 23:54:51.133301 28663 raft_consensus.cc:1495] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:54:51.133548 28663 raft_consensus_state.cc:377] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Trying to abort 3 pending transactions. W0628 23:54:51.133882 28982 catalog_manager.cc:2218] TS a19aaebe5ce646349f024b13675e09ed: delete failed for tablet a16b944a988b4388b55f8396b18aa8b9 with error code TABLET_NOT_RUNNING: Illegal state: State transition of tablet a16b944a988b4388b55f8396b18aa8b9 already in progress: deleting tablet I0628 23:54:51.134059 28982 catalog_manager.cc:2014] Scheduling retry of a16b944a988b4388b55f8396b18aa8b9 Delete Tablet RPC for TS=a19aaebe5ce646349f024b13675e09ed with a delay of 60ms (attempt = 1)... I0628 23:54:51.133669 28663 raft_consensus_state.cc:382] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Aborting transaction as it isn't in flight: id { term: 1 index: 3 } timestamp: 6009479540923101184 op_type: WRITE_OP write_request { tablet_id: "a16b944a988b4388b55f8396b18aa8b9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } row_operations { rows: "\001\007\000\223\227\r\000\3533Z|\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\025\311\006\006\312\204{\'\013\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\006\376J\036sKkM\026\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\272\222&?\271\021k}!\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\003\311X\000\"\344\364D,\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\214\227\255)\264}\270C7\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000(\327D\005\313\205\211jB\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\021\t\257j\317q|\tM\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\346\321\023I\205x94X\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\214~\027)g.uEc\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\321\2179\020\251\352\0203n\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000p\027\236\033:\310\024(y\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000dlel\324P\257v\204\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000+\313\343op\255/T\217\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000>\277#\006\230\202\340\024\232\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\335\323\235\037bb\3002\245\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\365){t\221\317\377A\260\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000qZ\224\001\206\261\2752\273\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000p\356\321C\331\035}\013\306\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000s\2272C|!u.\321\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\270k\351\013$\003\237\t\334\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000k5\207&\217\027\304v\347\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\0003\352\346C\030\335\031m\362\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\235\212\002=\265h\332q\375\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000x}\020:\020o\235\020\010\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\365\213\346O\316\252\356(\023\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000`\317\017M&\317\367H\036\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\0007\355@\005\222\270\225i)\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000e\257\210dsC\263I4\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\322X\021\030VO\336\032?\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\341\244*y\354\355\256^J\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\204Q\206.\370\310QtU\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000qMa%\343H#\027`\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\363:\206\rt.\210gk\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\306\366\241\035\\PPtv\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\254\003\314D}4\005.\201\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000%\022\271W!hV7\214\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\351j\316\t\006\354\314N\227\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000T\246\233o\n\016\305S\242\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000}\335\034.1Q\021i\255\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\332\235\351o\234\214zR\270\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\021\200\245kM\023{>\303\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000GD0\002p\222\322.\316\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\024\017\214\002Z\35319\331\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\013\177R|\361\352\022\014\344\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000hz\376-oY\026\036\357\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\327\225IM\345}\014\032\372\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\277R\035*2\217#k\005\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\221R\216k\301\344\316L\020\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000Lb\014#\216}\t\002\033\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello world" } external_consistency_mode: CLIENT_PROPAGATED } I0628 23:54:51.136464 28663 raft_consensus_state.cc:382] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Aborting transaction as it isn't in flight: id { term: 1 index: 4 } timestamp: 6009479540947378176 op_type: WRITE_OP write_request { tablet_id: "a16b944a988b4388b55f8396b18aa8b9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } row_operations { rows: "\001\007\000\341\032\016\000\316\251\006\036\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\310\"vG\037\235\345\034\013\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\013m\256&\004\001y\005\026\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000j\2601K\267%5+!\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\212\'?.\276\354<4,\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\275\310\332\013_\366\260H7\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000A\374\376Y\217=csB\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000w\262^x^\340\026\021M\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\025D\344t\247\37216X\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\275\005?\013\017\301\221Vc\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000/\354\031{\314@\341bn\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\313Ce2F\336E\026y\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\026\321\003E\277\036\224}\204\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\002\316n\007\035\351\226{\217\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000N\256\242p\247\274aH\232\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\021\235\250\010\207\313\337u\245\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\206H\2537\367q&N\260\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\00067\004>\t\332\304\006\273\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000W\210\306e\364;!M\306\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\2737\347@G\367\373\013\321\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000v\\\'K\206m\'\005\334\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\016{\207_!\005\2407\347\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\016\326\260j\332g\265\177\362\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\266Z\265^M\223VT\375\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000|\317\340\177{\027V\000\010\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000i\214\037\024\321K;#\023\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000i\230\356\t\227\032X\017\036\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000_\2679a\327\3575\024)\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\236\026\025a\335\020\200/4\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\2105\323\000$t_*?\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\0007\375\301`\201W\327_J\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000O\007\2622\211\355\375EU\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000E\341\365!n\212\230\025`\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\325F\250Q\324s\302~k\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000P\244B\020\207\2101\007v\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\301Z\372DHX\\\020\201\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\\\347\247\036\275XF!\214\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\\\361k\021\363\327\262E\227\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000D\266\177`\332[\227`\242\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\300v\017q=q=.\255\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\202s\325C\230~\222r\270\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\353g\265o\230\256\271n\303\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\362\265u`WT\374O\316\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\307D\005?#\205\346r\331\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\303\367.|\346b\214o\344\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000@#\261i\365t\201l\357\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000z\270%\'\252er\035\372\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\000\213pA\220\316\265C\005\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\252r\014U\205\"<$\020\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000X\\\376g\276\312`l\033\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello world" } external_consistency_mode: CLIENT_PROPAGATED } I0628 23:54:51.138736 28663 raft_consensus_state.cc:382] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Aborting transaction as it isn't in flight: id { term: 1 index: 5 } timestamp: 6009479540965576704 op_type: WRITE_OP write_request { tablet_id: "a16b944a988b4388b55f8396b18aa8b9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } row_operations { rows: "\001\007\000:\331\r\000\335n0\r\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\356u\276f\364\2200b\013\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\210\265|b;&ri\026\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\222!,E\270\033PT!\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000F\370KWp\350\230<,\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000$0\304Z\n\2724\0067\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\264\351\241o\255a\366nB\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\304\335\206q\026\251IMM\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\376\n\374\036\226\27155X\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000$B+Z\273w\003Nc\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\000\276\251E\273\025\371\nn\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\235\255\001g@S-\037y\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\275\236\264X\312\267!:\204\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\226L\251{GK\343\'\217\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\3066c;\237\037\241n\232\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000w8#\024\365\026P\024\245\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000>9\023\026\304 \023H\260\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\323H\314_\307E\301\\\273\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000d;\314\024\346,Ol\306\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\227\347\014Ba\2148]\321\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\027d\210+U8c\007\334\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000=X\007\003X\0162W\347\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000!\340K\027y\242gv\362\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\252\362\333\r\001~\030c\375\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000z\246\370\\E\303yH\010\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000/\014\0032O\373\024f\023\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\3443\177k\336\364\'l\036\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000KR=34\324\345~)\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\002\343\316\"(\252\231<4\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000-Gr\014\275\341\236\"?\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\014Q\366l\267\"C\037J\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000i,\234pA\333\'\035U\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000[\227\253#\250\351]V`\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\344@\227/$Q%sk\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\213M\362\026q\354\300}v\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\0007/\343\004b\3060_\201\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\300|0{o`N,\214\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\".\235M\375\341?\n\227\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000L\256\rh\3624.Z\242\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\037*\226\0177a\247K\255\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\256\210\337Y\232\205\206b\270\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\376s\255m\363`\232\026\303\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\034\375Rqcsg\177\316\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\355\251\310`?8\014\026\331\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000g\273@|\353\246\317}\344\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\324\316\327K2\347KE\357\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000(\2477z\307q\277[\372\001\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\337\356\306u\341\256lW\005\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000\236bM \243\203\2058\020\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000\001\007\000R_\205E&$57\033\002\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello worldhello world" } external_consistency_mode: CLIENT_PROPAGATED } I0628 23:54:51.140992 28663 raft_consensus.cc:1511] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:54:51.146298 28663 ts_tablet_manager.cc:955] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:54:51.150724 28663 ts_tablet_manager.cc:965] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: Tablet deleted. Last logged OpId: 1.5 I0628 23:54:51.151751 28663 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P 420c8e80c83b4919bd67708634ad5bd9: Deleted tablet blocks from disk I0628 23:54:51.153537 28982 catalog_manager.cc:2226] TS 420c8e80c83b4919bd67708634ad5bd9: tablet a16b944a988b4388b55f8396b18aa8b9 (table table-0 [id=8e56fdabbd3145cb936ad1912202dc42]) successfully deleted I0628 23:54:51.155433 28781 raft_consensus.cc:1495] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:54:51.155879 28781 raft_consensus.cc:1511] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:54:51.157229 28781 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.157331 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.158129 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] I0628 23:54:51.158623 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] W0628 23:54:51.159435 29001 raft_consensus.cc:578] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 LEADER]: Unable to take state lock to update committed index: Illegal state: Replica not in running state W0628 23:54:51.162967 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=-2147483648, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.163198 28974 batcher.cc:312] Not found: Failed to write batch of 50 ops to tablet a16b944a988b4388b55f8396b18aa8b9 after 3 attempt(s): GetTableLocations(table-0, int32 key=-2147483648, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.163673 28781 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.163763 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.164340 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init I0628 23:54:51.164806 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] ==28847==WARNING: Can't read from symbolizer at fd 28 W0628 23:54:51.165808 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=-2147483648, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.165969 28974 batcher.cc:312] Not found: Failed to write batch of 50 ops to tablet a16b944a988b4388b55f8396b18aa8b9 after 3 attempt(s): GetTableLocations(table-0, int32 key=-2147483648, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.166061 28781 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.166266 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.167649 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] I0628 23:54:51.168992 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==28730==WARNING: Can't read from symbolizer at fd 26 W0628 23:54:51.204618 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=684427710, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.205274 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=251883561, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.205957 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1038446407, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.206428 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1035083222, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.206908 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1215436899, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.207415 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=691594844, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.207842 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=152424324, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.208495 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=940978650, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.208382 28780 tablet_service.cc:653] Processing DeleteTablet for tablet a16b944a988b4388b55f8396b18aa8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:33865 W0628 23:54:51.209142 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=667744100, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.209604 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1108076640, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.210353 28982 catalog_manager.cc:2218] TS a19aaebe5ce646349f024b13675e09ed: delete failed for tablet a16b944a988b4388b55f8396b18aa8b9 with error code TABLET_NOT_RUNNING: Illegal state: State transition of tablet a16b944a988b4388b55f8396b18aa8b9 already in progress: deleting tablet I0628 23:54:51.210508 28982 catalog_manager.cc:2014] Scheduling retry of a16b944a988b4388b55f8396b18aa8b9 Delete Tablet RPC for TS=a19aaebe5ce646349f024b13675e09ed with a delay of 70ms (attempt = 2)... /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==28847==WARNING: Can't read from symbolizer at fd 28 W0628 23:54:51.224325 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2060830993, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.224833 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1251835078, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.225476 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1194226207, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.226197 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=601613223, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.226805 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1742431712, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.227349 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1441179343, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.227774 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=831055677, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.228350 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=205818086, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.228785 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1237812998, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.229552 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1701451229, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==28730==WARNING: Can't read from symbolizer at fd 26 W0628 23:54:51.230211 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1915086101, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.230954 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=635938264, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.231566 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1780822866, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==28847==WARNING: Can't read from symbolizer at fd 28 W0628 23:54:51.252792 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1379038672, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.254091 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1864905134, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.254757 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1457646016, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.255412 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=908804454, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.256008 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1804612780, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.256486 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1575861877, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.257161 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1151245089, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.257707 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1545583278, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.258327 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1326592057, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==28730==WARNING: Can't read from symbolizer at fd 26 W0628 23:54:51.258746 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1673285787, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.259368 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2031676545, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.259886 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=758871746, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.260512 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2043652719, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.261090 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2046944184, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.261703 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=837578005, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.262292 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=893091846, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.263052 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=375951019, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.263783 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1051041776, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==28730==WARNING: external symbolizer didn't start up correctly! ==28730==WARNING: Can't write to symbolizer at fd 29 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==28847==WARNING: Can't read from symbolizer at fd 28 ==28847==WARNING: Failed to use and restart external symbolizer! W0628 23:54:51.279079 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=67725732, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC ================== WARNING: ThreadSanitizer: data race (pid=28847) Write of size 4 at 0x7d700006f8f0 by thread T127: #0 (libtablet.so+0x000000299980) #1 (libtablet.so+0x000000298fc4) W0628 23:54:51.279662 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2120976944, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T128: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T127: #0 (kudu-tserver+0x000000440858) ==28730==WARNING: Failed to use and restart external symbolizer! I0628 23:54:51.284153 28780 tablet_service.cc:653] Processing DeleteTablet for tablet a16b944a988b4388b55f8396b18aa8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:33865 ================== WARNING: ThreadSanitizer: data race (pid=28730) Atomic write of size 4 at 0x7d700006f8f0 by thread T122: W0628 23:54:51.284755 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=990215854, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #0 (kudu-tserver+0x000000466be0) W0628 23:54:51.285353 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1967980151, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.286098 28982 catalog_manager.cc:2218] TS a19aaebe5ce646349f024b13675e09ed: delete failed for tablet a16b944a988b4388b55f8396b18aa8b9 with error code TABLET_NOT_RUNNING: Illegal state: State transition of tablet a16b944a988b4388b55f8396b18aa8b9 already in progress: deleting tablet I0628 23:54:51.286267 28982 catalog_manager.cc:2014] Scheduling retry of a16b944a988b4388b55f8396b18aa8b9 Delete Tablet RPC for TS=a19aaebe5ce646349f024b13675e09ed with a delay of 90ms (attempt = 3)... #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T123: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) W0628 23:54:51.287822 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=471279105, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) W0628 23:54:51.288311 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=369784467, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29015, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #1 (libkudu_util.so+0x0000001d048f) #4 (libkudu_util.so+0x000000224902) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x000000224704) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #6 (libkudu_util.so+0x000000224632) #8 (libtablet.so+0x000000295539) #7 (libtablet.so+0x00000023ae76) #9 (libtablet.so+0x0000002988dc) #8 (libtablet.so+0x00000023908d) #10 (libtablet.so+0x000000297af3) #9 (libtablet.so+0x000000234615) #11 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x000000234508) #12 (libtablet.so+0x0000002965d3) #11 (libtablet.so+0x00000023444a) #13 (libtablet.so+0x00000028cafd) #12 (libmaster_rpc.so+0x000000011c31) #14 (libtablet.so+0x00000028936b) #13 (libconsensus.so+0x000000096280) #15 (libtablet.so+0x0000001dcebd) #14 (libconsensus.so+0x0000000f0f2d) #16 (libtablet.so+0x0000001dde5d) #15 (libconsensus.so+0x0000000d93bb) #17 (libtablet.so+0x0000001ddcaf) #16 (libconsensus.so+0x0000000d8c4f) #18 (libtablet.so+0x0000002438f1) #17 (libtserver.so+0x00000012dafc) #19 (libtablet.so+0x00000023b2f5) #18 (libconsensus_proto.so+0x00000006d374) #20 (libtablet.so+0x00000023d2c6) #19 (libconsensus_proto.so+0x00000006d18e) #21 (libtablet.so+0x00000023d255) #20 (libkrpc.so+0x000000121d48) #22 (libtablet.so+0x00000023d1ea) #21 (libkrpc.so+0x000000121948) #23 (libconsensus.so+0x0000000befdd) #22 (libkrpc.so+0x00000012369d) #24 (libconsensus.so+0x0000000c5726) #23 (libkrpc.so+0x000000125476) #25 (libconsensus.so+0x0000000c568e) #24 (libkrpc.so+0x0000001253da) #26 (libconsensus.so+0x0000000c5643) #25 (libkrpc.so+0x000000125383) W0628 23:54:51.289566 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=149875028, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #27 (libconsensus.so+0x0000000c5401) #26 (libkrpc.so+0x0000001251a9) #28 (libkrpc.so+0x0000000cce71) #27 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #28 (libkudu_util.so+0x00000021b55f) #30 (libkudu_util.so+0x000000224fcf) #29 (kudu-tserver+0x000000421829) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) ================== #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29010, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) W0628 23:54:51.290041 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1887379673, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) I0628 23:54:51.290235 28781 transaction_tracker.cc:230] Dumping currently running transactions: #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29011, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299980) ================== W0628 23:54:51.290510 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2041253271, 2) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.291111 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=838313916, 4) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.290356 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.291167 28898 consensus_peers.cc:351] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 -> Peer 420c8e80c83b4919bd67708634ad5bd9 (127.106.77.0:44123): Closing peer: 420c8e80c83b4919bd67708634ad5bd9 W0628 23:54:51.291528 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1593722554, 3) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.291283 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] I0628 23:54:51.291820 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] W0628 23:54:51.292321 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1907796873, 4) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.293081 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1684210850, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.293548 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=390369058, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.293692 28781 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.293772 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.294539 28898 raft_consensus.cc:1511] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781 [term 1 LEADER]: Raft consensus is shut down! I0628 23:54:51.294425 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] I0628 23:54:51.294883 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] I0628 23:54:51.296802 28898 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.296970 28781 transaction_tracker.cc:230] Dumping currently running transactions: W0628 23:54:51.297528 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=963900988, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.297101 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] W0628 23:54:51.297984 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=471752782, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.296900 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d440006dd80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] W0628 23:54:51.298532 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1851514186, 3) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.298288 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400063d80 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] W0628 23:54:51.298985 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=498456115, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.297721 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] ================== WARNING: ThreadSanitizer: data race (pid=28847) Write of size 8 at 0x7d700006f8e0 by thread T127: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8e0 by thread T128: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T127: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) ================== WARNING: ThreadSanitizer: data race (pid=28730) #18 (libtablet.so+0x0000002438f1) Write of size 8 at 0x7d700006f838 by thread T124: #19 (libtablet.so+0x00000023b2f5) #0 (libtablet.so+0x00000028bc27) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #1 (libtablet.so+0x000000298a34) #23 (libconsensus.so+0x0000000befdd) #2 (libtablet.so+0x0000002979d1) #24 (libconsensus.so+0x0000000c5726) #3 (libtablet.so+0x000000296aeb) #25 (libconsensus.so+0x0000000c568e) #4 (libtablet.so+0x0000002965d3) #26 (libconsensus.so+0x0000000c5643) #5 (libtablet.so+0x00000028cafd) #27 (libconsensus.so+0x0000000c5401) #6 (libtablet.so+0x00000028936b) #28 (libkrpc.so+0x0000000cce71) #7 (libtablet.so+0x0000001dcebd) #29 (libkudu_util.so+0x000000226bcd) #8 (libtablet.so+0x0000001dde5d) #30 (libkudu_util.so+0x000000224fcf) #9 (libtablet.so+0x0000001ddcaf) #31 (libkudu_util.so+0x00000022941e) #10 (libtablet.so+0x0000002438f1) #32 (libkudu_util.so+0x00000022937b) #11 (libtablet.so+0x00000023b2f5) #33 (libkudu_util.so+0x000000229303) #12 (libtablet.so+0x00000023d2c6) #34 (libkudu_util.so+0x000000229101) #13 (libtablet.so+0x00000023d255) #35 (libkrpc.so+0x0000000cce71) #14 (libtablet.so+0x00000023d1ea) #36 (libkudu_util.so+0x00000021b55f) #15 (libconsensus.so+0x0000000befdd) #37 (kudu-tserver+0x000000421829) #16 (libconsensus.so+0x0000000c5726) Thread T127 'apply [worker]-' #17 (libconsensus.so+0x0000000c568e) (tid=29010, running) created by thread T125 at: #18 (libconsensus.so+0x0000000c5643) #0 (kudu-tserver+0x00000044a123) #19 (libconsensus.so+0x0000000c5401) #1 (libkudu_util.so+0x00000021ade7) #20 (libkrpc.so+0x0000000cce71) #2 (libkudu_util.so+0x0000002262ae) #21 (libkudu_util.so+0x000000226bcd) #3 (libkudu_util.so+0x000000224304) #22 (libkudu_util.so+0x000000224fcf) #4 (libkudu_util.so+0x000000224902) #23 (libkudu_util.so+0x00000022941e) #5 (libkudu_util.so+0x000000224704) #24 (libkudu_util.so+0x00000022937b) #6 (libkudu_util.so+0x000000224632) #25 (libkudu_util.so+0x000000229303) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #26 (libkudu_util.so+0x000000229101) #9 (libtablet.so+0x000000234615) #27 (libkrpc.so+0x0000000cce71) #10 (libtablet.so+0x000000234508) #28 (libkudu_util.so+0x00000021b55f) #11 (libtablet.so+0x00000023444a) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f839 by thread T123 #12 (libmaster_rpc.so+0x000000011c31) : #0 (libtablet.so+0x0000001fc02e) #13 (libconsensus.so+0x000000096280) #1 (libtablet.so+0x0000001fbf59) #14 (libconsensus.so+0x0000000f0f2d) #2 (libtablet.so+0x0000001fbbd2) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #3 (libtablet.so+0x0000001fb7ea) #17 (libconsensus.so+0x0000000d73d0) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #18 (libconsensus.so+0x0000000d79cd) #6 (libtablet.so+0x0000001fad7b) #19 (libconsensus.so+0x0000000a97f2) #7 (libtablet.so+0x000000295cf2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #8 (libtablet.so+0x00000028c6b7) #22 (libconsensus.so+0x0000000ba4d3) #9 (libtablet.so+0x000000289219) #23 (libconsensus.so+0x0000000befdd) #10 (libtablet.so+0x0000001dcebd) #24 (libconsensus.so+0x0000000c5726) #11 (libtablet.so+0x0000001dde5d) W0628 23:54:51.300379 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=601888258, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #25 (libconsensus.so+0x0000000c568e) #12 (libtablet.so+0x0000001ddcaf) #26 (libconsensus.so+0x0000000c5643) #13 (libtablet.so+0x0000002438f1) #27 (libconsensus.so+0x0000000c5401) #14 (libtablet.so+0x00000023b2f5) #28 (libkrpc.so+0x0000000cce71) #15 (libtablet.so+0x00000023d2c6) #29 (libkudu_util.so+0x000000226bcd) #16 (libtablet.so+0x00000023d255) #30 (libkudu_util.so+0x000000224fcf) #17 (libtablet.so+0x00000023d1ea) #31 (libkudu_util.so+0x00000022941e) #18 (libconsensus.so+0x0000000befdd) #32 (libkudu_util.so+0x00000022937b) #19 (libconsensus.so+0x0000000c5726) #33 (libkudu_util.so+0x000000229303) #20 (libconsensus.so+0x0000000c568e) #34 (libkudu_util.so+0x000000229101) #21 (libconsensus.so+0x0000000c5643) #35 (libkrpc.so+0x0000000cce71) #22 (libconsensus.so+0x0000000c5401) #36 (libkudu_util.so+0x00000021b55f) #23 (libkrpc.so+0x0000000cce71) #37 (kudu-tserver+0x000000421829) #24 (libkudu_util.so+0x000000226bcd) Thread T128 #25 (libkudu_util.so+0x000000224fcf) 'apply [worker]-' (tid=29011, running) created by thread T125 at: #26 (libkudu_util.so+0x00000022941e) #0 (kudu-tserver+0x00000044a123) #27 (libkudu_util.so+0x00000022937b) #1 (libkudu_util.so+0x00000021ade7) #28 (libkudu_util.so+0x000000229303) #2 (libkudu_util.so+0x0000002262ae) #29 (libkudu_util.so+0x000000229101) #3 (libkudu_util.so+0x000000224304) #30 (libkrpc.so+0x0000000cce71) #4 (libkudu_util.so+0x000000224902) #31 (libkudu_util.so+0x00000021b55f) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #32 (kudu-tserver+0x000000421829) #7 (libtablet.so+0x00000023ae76) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #8 (libtablet.so+0x00000023908d) #0 (kudu-tserver+0x000000440858) #9 (libtablet.so+0x000000234615) #1 (libkudu_util.so+0x0000001d048f) #10 (libtablet.so+0x000000234508) #2 (libkudu_util.so+0x0000001d039a) #11 (libtablet.so+0x00000023444a) #3 (libkudu_util.so+0x0000001d11a9) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #4 (libkudu_util.so+0x0000001d0e5f) #14 (libconsensus.so+0x0000000f0f2d) #5 (libkudu_util.so+0x0000001cc3ab) #15 (libconsensus.so+0x0000000f0017) #6 (libkudu_util.so+0x0000001cb1d4) #16 (libconsensus.so+0x0000000d76b1) #7 (libkudu_util.so+0x0000001cba25) #17 (libconsensus.so+0x0000000d73d0) #8 (libtablet.so+0x000000295539) #18 (libconsensus.so+0x0000000d79cd) #9 (libtablet.so+0x0000002988dc) #19 (libconsensus.so+0x0000000a97f2) #10 (libtablet.so+0x000000297af3) #20 (libconsensus.so+0x0000000ba5e8) #11 (libtablet.so+0x000000296aeb) #21 (libconsensus.so+0x0000000ba558) #12 (libtablet.so+0x0000002965d3) #22 (libconsensus.so+0x0000000ba4d3) #13 (libtablet.so+0x00000028cafd) #23 (libconsensus.so+0x0000000befdd) #14 (libtablet.so+0x00000028936b) #24 (libconsensus.so+0x0000000c5726) #15 (libtablet.so+0x0000001dcebd) #25 (libconsensus.so+0x0000000c568e) #16 (libtablet.so+0x0000001dde5d) #26 (libconsensus.so+0x0000000c5643) #17 (libtablet.so+0x0000001ddcaf) #27 (libconsensus.so+0x0000000c5401) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #28 (libkrpc.so+0x0000000cce71) #23 (libconsensus.so+0x0000000befdd) #29 (libkudu_util.so+0x000000226bcd) #24 (libconsensus.so+0x0000000c5726) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #25 (libconsensus.so+0x0000000c568e) #32 (libkudu_util.so+0x00000022937b) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #33 (libkudu_util.so+0x000000229303) #28 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) #33 (libkudu_util.so+0x000000229303) ================== #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=29017, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) W0628 23:54:51.300815 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=782179377, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) I0628 23:54:51.300902 28898 transaction_tracker.cc:230] Dumping currently running transactions: #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29015, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) W0628 23:54:51.301254 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2007628948, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) W0628 23:54:51.301834 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1432452813, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) I0628 23:54:51.301038 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d440006dd80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== W0628 23:54:51.302458 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=774513665, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC ================== WARNING: ThreadSanitizer: data race (pid=28730) Atomic write of size 8 at 0x7d700006f830 by thread T124: #0 (kudu-tserver+0x000000466ec0) I0628 23:54:51.302240 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400063d80 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] W0628 23:54:51.302917 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1786278613, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f831 by thread T123: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=29017, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) ================== #6 (libkudu_util.so+0x000000224632) WARNING: ThreadSanitizer: data race (pid=28847) Write of size 8 at 0x7d700006f838 by thread T127: #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #0 (libtablet.so+0x00000028bc27) #9 (libtablet.so+0x000000234615) #1 (libtablet.so+0x000000298a34) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #2 (libtablet.so+0x000000297d2b) #12 (libmaster_rpc.so+0x000000011c31) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #13 (libconsensus.so+0x000000096280) #5 (libtablet.so+0x00000028cafd) #14 (libconsensus.so+0x0000000f0f2d) #6 (libtablet.so+0x00000028936b) #15 (libconsensus.so+0x0000000d93bb) #7 (libtablet.so+0x0000001dcebd) #16 (libconsensus.so+0x0000000d8c4f) #8 (libtablet.so+0x0000001dde5d) #17 (libtserver.so+0x00000012dafc) #9 (libtablet.so+0x0000001ddcaf) #18 (libconsensus_proto.so+0x00000006d374) #10 (libtablet.so+0x0000002438f1) #19 (libconsensus_proto.so+0x00000006d18e) #11 (libtablet.so+0x00000023b2f5) #20 (libkrpc.so+0x000000121d48) #12 (libtablet.so+0x00000023d2c6) #21 (libkrpc.so+0x000000121948) #13 (libtablet.so+0x00000023d255) #22 (libkrpc.so+0x00000012369d) #14 (libtablet.so+0x00000023d1ea) #23 (libkrpc.so+0x000000125476) #15 (libconsensus.so+0x0000000befdd) #24 (libkrpc.so+0x0000001253da) #16 (libconsensus.so+0x0000000c5726) #25 (libkrpc.so+0x000000125383) #17 (libconsensus.so+0x0000000c568e) #26 (libkrpc.so+0x0000001251a9) #18 (libconsensus.so+0x0000000c5643) #27 (libkrpc.so+0x0000000cce71) #19 (libconsensus.so+0x0000000c5401) #28 (libkudu_util.so+0x00000021b55f) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29015, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #22 (libkudu_util.so+0x000000224fcf) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #23 (libkudu_util.so+0x00000022941e) #26 (libkrpc.so+0x0000001251a9) #24 (libkudu_util.so+0x00000022937b) #27 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x00000021b55f) #26 (libkudu_util.so+0x000000229101) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' #27 (libkrpc.so+0x0000000cce71) (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #28 (libkudu_util.so+0x00000021b55f) #1 (libkudu_util.so+0x00000021ade7) #29 (kudu-tserver+0x000000421829) #2 (libkudu_util.so+0x0000002262ae) Previous read of size 1 at 0x7d700006f83a by thread T128: #3 (libkudu_util.so+0x000000224304) #0 (libtablet.so+0x0000001fc02e) #4 (libkudu_util.so+0x000000224902) #1 (libtablet.so+0x0000001fbf59) #5 (libkudu_util.so+0x000000224704) #2 (libtablet.so+0x0000001fbbd2) #6 (libkudu_util.so+0x000000224632) #3 (libtablet.so+0x0000001fb7ea) #7 (libtablet.so+0x00000023ae76) #4 (libtablet.so+0x0000001fb609) #8 (libtablet.so+0x00000023908d) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #9 (libtablet.so+0x000000234615) #7 (libtablet.so+0x000000295cf2) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #8 (libtablet.so+0x00000028c6b7) #12 (libmaster_rpc.so+0x000000011c31) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #13 (libconsensus.so+0x000000096280) #11 (libtablet.so+0x0000001dde5d) #14 (libconsensus.so+0x0000000f0f2d) #12 (libtablet.so+0x0000001ddcaf) #15 (libconsensus.so+0x0000000d93bb) #13 (libtablet.so+0x0000002438f1) #16 (libconsensus.so+0x0000000d8c4f) #14 (libtablet.so+0x00000023b2f5) #17 (libtserver.so+0x00000012dafc) #15 (libtablet.so+0x00000023d2c6) #18 (libconsensus_proto.so+0x00000006d374) #16 (libtablet.so+0x00000023d255) #19 (libconsensus_proto.so+0x00000006d18e) #17 (libtablet.so+0x00000023d1ea) #20 (libkrpc.so+0x000000121d48) #18 (libconsensus.so+0x0000000befdd) #21 (libkrpc.so+0x000000121948) #19 (libconsensus.so+0x0000000c5726) #22 (libkrpc.so+0x00000012369d) #20 (libconsensus.so+0x0000000c568e) #23 (libkrpc.so+0x000000125476) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000001253da) #23 (libkrpc.so+0x0000000cce71) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #24 (libkudu_util.so+0x000000226bcd) #27 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000021b55f) #26 (libkudu_util.so+0x00000022941e) W0628 23:54:51.304271 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1377560359, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #29 (kudu-tserver+0x000000421829) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) ================== #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T127: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29010, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) W0628 23:54:51.304684 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=832931942, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29011, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) ================== #26 (libconsensus.so+0x0000000c5643) WARNING: ThreadSanitizer: data race (pid=28730) Write of size 8 at 0x7d700006f8a8 by thread T124: #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== #0 (libtablet.so+0x000000298ab0) W0628 23:54:51.305090 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2096822494, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #1 (libtablet.so+0x0000002979d1) I0628 23:54:51.305124 28898 transaction_tracker.cc:230] Dumping currently running transactions: #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) I0628 23:54:51.301595 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] #4 (libtablet.so+0x00000028cafd) W0628 23:54:51.305511 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1083375467, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.305230 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d440006dd80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] #5 (libtablet.so+0x00000028936b) W0628 23:54:51.305938 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=616319524, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8a8 by thread T123: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) W0628 23:54:51.306429 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=357906892, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) I0628 23:54:51.305979 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400063d80 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=29017, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29015, running) created by thread T69 at: W0628 23:54:51.306849 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=23808242, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== W0628 23:54:51.307265 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=717614745, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC ================== WARNING: ThreadSanitizer: data race (pid=28847) Write of size 8 at 0x7d700006f828 by thread T127: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x000000297d2b) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) I0628 23:54:51.307343 28781 transaction_tracker.cc:230] Dumping currently running transactions: #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f828 by thread T128: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) W0628 23:54:51.307701 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1062285619, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T127: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29010, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) W0628 23:54:51.308109 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1635228998, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) I0628 23:54:51.307492 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] Thread T128 'apply [worker]-' (tid=29011, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== I0628 23:54:51.308540 28898 transaction_tracker.cc:230] Dumping currently running transactions: W0628 23:54:51.308552 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1422149660, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.308997 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=660808427, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.309464 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=689216461, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.308281 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] I0628 23:54:51.308672 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d440006dd80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] W0628 23:54:51.309876 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=636132473, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.309617 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400063d80 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] W0628 23:54:51.310329 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=997425451, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.310781 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1409964817, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC ================== WARNING: ThreadSanitizer: data race (pid=28847) Write of size 8 at 0x7d700006f8b0 by thread T127: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x000000297d2b) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8b0 by thread T128: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) W0628 23:54:51.311265 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1230062268, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T127: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29010, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) W0628 23:54:51.311687 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=373422817, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29011, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== W0628 23:54:51.312114 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=71559264, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.312144 28898 transaction_tracker.cc:230] Dumping currently running transactions: W0628 23:54:51.312585 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1470931600, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.312276 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d440006dd80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] W0628 23:54:51.313029 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2095346907, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.313459 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=632934269, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.313012 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400063d80 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] W0628 23:54:51.313911 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1938748256, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.314376 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1876782457, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.314636 28898 transaction_tracker.cc:230] Dumping currently running transactions: W0628 23:54:51.314787 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=175137621, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC ================== WARNING: ThreadSanitizer: data race (pid=28730) Write of size 8 at 0x7d700006f858 by thread T122: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) W0628 23:54:51.315224 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=696798383, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f858 by thread T123: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) I0628 23:54:51.314759 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d440006dd80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) W0628 23:54:51.315673 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1158362684, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) W0628 23:54:51.316112 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1350260852, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.309603 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] I0628 23:54:51.315536 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400063d80 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] W0628 23:54:51.316551 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1057114357, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.316980 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1832953977, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.317391 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1462468809, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.317649 28898 transaction_tracker.cc:230] Dumping currently running transactions: W0628 23:54:51.317793 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1502281231, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.318297 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=817669927, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29015, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=28730) Write of size 8 at 0x7d700006f850 by thread T122: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f851 by thread T123: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29015, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init I0628 23:54:51.317751 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d440006dd80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.324405 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400063d80 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] ================== WARNING: ThreadSanitizer: data race (pid=28730) Atomic write of size 8 at 0x7d700006f818 by thread T122: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f819 by thread T123: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29015, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=28730) Write of size 8 at 0x7d700006f8d0 by thread T122: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8d0 by thread T123: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29015, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) ================== WARNING: ThreadSanitizer: data race (pid=28847) Write of size 8 at 0x7d700006f808 by thread T127: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f808 by thread T128: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T127: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29010, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29011, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) ================== I0628 23:54:51.330137 28898 transaction_tracker.cc:230] Dumping currently running transactions: W0628 23:54:51.332077 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=660014299, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.332777 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1412506919, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== I0628 23:54:51.334681 28781 transaction_tracker.cc:230] Dumping currently running transactions: ================== WARNING: ThreadSanitizer: data race (pid=28847) Read of size 8 at 0x7d500003fd38 by thread T127: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) I0628 23:54:51.334774 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500003fd38 by thread T128: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29010, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) I0628 23:54:51.335319 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29011, running) created by thread T125 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=28984, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== I0628 23:54:51.337349 28781 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.337432 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.337981 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] I0628 23:54:51.340358 28781 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.330286 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d440006dd80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.340446 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] ================== WARNING: ThreadSanitizer: data race (pid=28730) Write of size 8 at 0x7d700006f808 by thread T124: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) I0628 23:54:51.340911 28898 transaction_tracker.cc:232] RD-P WriteTransaction [type=LEADER, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400063d80 [op_id=(term: 1 index: 4), ts=6009479540947378176, rows=[INSERT (int32 key=924385, int32 int_val=503753166, string string_val=hello world), INSERT (int32 key=1198924488, int32 int_val=484810015, string string_val=hello world), INSERT (int32 key=648965387, int32 int_val=91816196, string string_val=hello world), ...]]] #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) I0628 23:54:51.340972 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f808 by thread T122: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=29017, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) ================== ================== WARNING: ThreadSanitizer: data race (pid=28730) Write of size 8 at 0x7d500003fd38 by thread T124: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500003fd38 by thread T122: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=29017, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=29014, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=28981, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== I0628 23:54:51.344168 28781 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.344281 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.344851 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] I0628 23:54:51.347553 28898 ts_tablet_manager.cc:955] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:54:51.348361 28781 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:54:51.348451 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400057f80 [op_id=(term: 1 index: 5), ts=6009479540965576704, rows=[INSERT (int32 key=907578, int32 int_val=221277917, string string_val=hello world), INSERT (int32 key=1723758062, int32 int_val=1647350004, string string_val=hello world), INSERT (int32 key=1652340104, int32 int_val=1769088571, string string_val=hello world), ...]]] I0628 23:54:51.348968 28781 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:54:51, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 3), ts=6009479540923101184, rows=[INSERT (int32 key=890771, int32 int_val=2086286315, string string_val=hello world), INSERT (int32 key=101107989, int32 int_val=662406346, string string_val=hello world), INSERT (int32 key=508231174, int32 int_val=1298877299, string string_val=hello world), ...]]] I0628 23:54:51.352454 28898 ts_tablet_manager.cc:965] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: Tablet deleted. Last logged OpId: 1.5 I0628 23:54:51.353585 28898 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P 7333285db9804dde9478033cabd5d781: Deleted tablet blocks from disk F0628 23:54:51.354387 28898 mem_tracker.cc:186] Check failed: consumption() == 0 Memory tracker txn_tracker->tablet-a16b944a988b4388b55f8396b18aa8b9->server->root has unreleased consumption 2400 *** Check failure stack trace: *** I0628 23:54:51.357610 28781 ts_tablet_manager.cc:955] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: Deleting tablet data with delete state TABLET_DATA_DELETED @ 0x7f378e217304 google::LogMessage::SendToLog() at ??:0 I0628 23:54:51.362042 28781 ts_tablet_manager.cc:965] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: Tablet deleted. Last logged OpId: 1.5 @ 0x7f378e217800 google::LogMessage::Flush() at ??:0 I0628 23:54:51.363095 28781 tablet_bootstrap.cc:393] T a16b944a988b4388b55f8396b18aa8b9 P a19aaebe5ce646349f024b13675e09ed: Deleted tablet blocks from disk @ 0x7f378e21b432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f378ef05098 kudu::MemTracker::~MemTracker() at ??:0 @ 0x7f378ef087ec std::_Sp_counted_ptr<>::_M_dispose() at ??:0 @ 0x7f3795d8cc01 std::_Sp_counted_base<>::_M_release() at ??:0 I0628 23:54:51.378146 28781 tablet_service.cc:653] Processing DeleteTablet for tablet a16b944a988b4388b55f8396b18aa8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:54:51 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:33865 W0628 23:54:51.378926 28982 catalog_manager.cc:2207] TS a19aaebe5ce646349f024b13675e09ed: delete failed for tablet a16b944a988b4388b55f8396b18aa8b9 because the tablet was not found. No further retry: Not found: Tablet not found: a16b944a988b4388b55f8396b18aa8b9 @ 0x7f3795d8cbb7 std::__shared_count<>::~__shared_count() at ??:0 @ 0x7f3795dc0f5e std::__shared_ptr<>::~__shared_ptr() at ??:0 W0628 23:54:51.386916 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1934601517, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.389067 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1040657745, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f379486dd11 kudu::tablet::TransactionTracker::~TransactionTracker() at ??:0 W0628 23:54:51.390197 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2128918906, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.390935 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=67450846, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.391803 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=195904909, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.392340 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=893886335, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f379485a60b kudu::tablet::TabletPeer::~TabletPeer() at ??:0 W0628 23:54:51.393218 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=767298758, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.393803 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1014296435, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f379485a7ba kudu::tablet::TabletPeer::~TabletPeer() at ??:0 @ 0x7f3795da1998 kudu::RefCountedThreadSafe<>::DeleteInternal() at ??:0 @ 0x7f3795da195a kudu::DefaultRefCountedThreadSafeTraits<>::Destruct() at ??:0 W0628 23:54:51.406167 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1248544392, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.407863 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1550986359, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f3795da192a kudu::RefCountedThreadSafe<>::Release() at ??:0 W0628 23:54:51.409469 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=221368586, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.410079 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=358243816, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f3795d9ed4b scoped_refptr<>::~scoped_refptr() at ??:0 W0628 23:54:51.411703 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=370671631, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.412436 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1007513197, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f3795de7494 kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 W0628 23:54:51.415642 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=984471174, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.416697 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1543323806, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.418632 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1782267541, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f3795dc4497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 W0628 23:54:51.419102 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=477814799, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.419651 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1723423404, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.420292 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=512449146, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.421164 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1280601202, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.421979 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=430988085, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f3793982ef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 W0628 23:54:51.423486 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=895412472, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.424275 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1298282795, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.425317 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=91664856, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f3793982d0f std::_Function_handler<>::_M_invoke() at ??:0 W0628 23:54:51.426314 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=877680520, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.428969 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=638111842, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f37934b3d49 std::function<>::operator()() at ??:0 W0628 23:54:51.429464 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=611263118, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.430222 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1106416276, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.430801 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1526085521, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f37934b3949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 W0628 23:54:51.433835 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2039361018, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.434525 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1144857543, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.436332 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2029887091, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f37934b569e kudu::rpc::ServicePool::RunThread() at ??:0 W0628 23:54:51.436914 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=165048524, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.438535 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=280259533, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.439316 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1682025777, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f37934b7477 boost::_mfi::mf0<>::operator()() at ??:0 W0628 23:54:51.440119 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1116030698, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.440795 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=657925823, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.442160 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1105511579, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.442729 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1183801925, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.443331 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1864701262, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f37934b73db boost::_bi::list1<>::operator()<>() at ??:0 W0628 23:54:51.443989 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1748314483, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.445713 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=155656527, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.446344 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=583314618, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f37934b7384 boost::_bi::bind_t<>::operator()() at ??:0 W0628 23:54:51.447942 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1381467524, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.448824 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1070684722, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.450495 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=280377278, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f37934b71aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 W0628 23:54:51.451138 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2134000940, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.451709 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1166164399, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.452219 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=939239766, 2) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.452911 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1220280519, 3) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.453424 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=847700407, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f379345ee72 boost::function0<>::operator()() at ??:0 W0628 23:54:51.455772 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=663676007, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.456356 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2038552507, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f378ef4d560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 W0628 23:54:51.457238 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=904652986, 2) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.457871 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2034800467, 4) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.458819 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1094074329, 4) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.459542 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1718174671, 1) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f379098c182 start_thread at ??:0 W0628 23:54:51.462324 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=962428601, 4) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.462977 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1388249752, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ 0x7f378c51b47d clone at ??:0 W0628 23:54:51.464282 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2081274013, 2) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.464735 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1554428785, 3) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.465486 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=990672322, 4) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.466106 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1466747867, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC @ (nil) (unknown) W0628 23:54:51.467684 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1056459492, 4) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.468142 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1143019205, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.468883 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=157677444, 4) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.469473 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1809161724, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.470216 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=982168219, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.470722 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1858271902, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.472486 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=240030486, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.472936 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1596840739, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.474103 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=698382666, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.474512 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=19296878, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.476305 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1088306231, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.476783 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=861949449, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.477535 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2098899014, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.478301 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1276331686, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.478863 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1262801180, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.479399 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1170038851, 6) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.479866 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=976243359, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.480533 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1790988478, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.481358 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1331251941, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.482082 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=345051459, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.482635 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2117452708, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.483309 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1375271519, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.483916 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1203954443, 5) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.484558 28982 catalog_manager.cc:1957] TS 7333285db9804dde9478033cabd5d781: Delete Tablet RPC failed for tablet a16b944a988b4388b55f8396b18aa8b9: Network error: Recv() got EOF from remote (error 108) I0628 23:54:51.484719 28982 catalog_manager.cc:2014] Scheduling retry of a16b944a988b4388b55f8396b18aa8b9 Delete Tablet RPC for TS=7333285db9804dde9478033cabd5d781 with a delay of 35ms (attempt = 1)... W0628 23:54:51.485569 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1370680503, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.486331 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=15816314, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.487054 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1265900889, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.487705 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1997902774, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.489382 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1805030940, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.490095 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=634075201, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.490753 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1078276565, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.491327 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=123799313, 7) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.491801 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1877553144, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.492539 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=284625577, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.493158 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1715948825, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.493574 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=409586663, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.494452 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1629296411, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.494956 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=721059414, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.495683 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=787505927, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.496350 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=40878135, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.496918 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1253378054, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.497473 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1798372635, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.498142 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=149356730, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.498884 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1184806792, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.499559 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1675975364, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.500244 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=576445334, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.501121 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=979195746, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.501626 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1511743506, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.502341 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1043154756, 8) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.502903 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1847461760, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.503495 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1199867971, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.504147 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1730149248, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.504637 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1254885218, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.505481 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=524307239, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.506105 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1961679452, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.506703 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1974456745, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.507232 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1861766014, 9) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.507838 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=405631693, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.508654 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=824225921, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.509250 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=623352468, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.509775 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1520585177, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.510500 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1059044341, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.511137 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1401571527, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.511564 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2061880212, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.512061 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=944544640, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.512630 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1820105027, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.513046 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1242542791, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.513623 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2000798343, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.514318 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=802820271, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.515197 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=885363128, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.516001 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=382702699, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.516556 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=175621942, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.517163 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=472035598, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.517597 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1515162300, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.518077 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=646132222, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.518599 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=27231973, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.519172 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=30955291, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.519760 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1165825749, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.520371 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1684671754, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.520970 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=779532112, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.521517 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=969201683, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.522122 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1771586005, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.522640 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1413619031, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.523181 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1971722839, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.523569 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=49317315, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.523994 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1321953440, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.524744 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=2125144382, 15) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.525318 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=347846076, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.526180 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=810847341, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.526744 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=462501901, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.527531 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=9596065, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.528045 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1320328679, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.528587 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1710858200, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.529072 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1120297285, 10) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.529438 28982 catalog_manager.cc:1957] TS 7333285db9804dde9478033cabd5d781: Delete Tablet RPC failed for tablet a16b944a988b4388b55f8396b18aa8b9: Network error: Client connection negotiation failed: client connection to 127.106.77.2:40485: connect: Connection refused (error 111) W0628 23:54:51.529541 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1927357459, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.529602 28982 catalog_manager.cc:2014] Scheduling retry of a16b944a988b4388b55f8396b18aa8b9 Delete Tablet RPC for TS=7333285db9804dde9478033cabd5d781 with a delay of 75ms (attempt = 2)... W0628 23:54:51.530093 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=823463231, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.530578 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1128525053, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.531021 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1625446854, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.531452 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=211473973, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.531869 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=180776675, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.532335 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=354287670, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.532747 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1026918078, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.533277 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=284144267, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.533692 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1498917492, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.534118 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=350399817, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.534518 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1258468377, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.534927 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=253603252, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.535356 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=66271999, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.535778 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1650081501, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.536238 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1606496496, 15) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.536716 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1754614709, 15) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.537212 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=68486442, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.537631 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=4164025, 15) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.538069 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1690528634, 11) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.538529 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1355699354, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.538923 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=407745777, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.539367 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1859430803, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.539758 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=352801767, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.540292 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1396269226, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.540709 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1227059848, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.541116 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1079416974, 15) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.541558 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1529936956, 12) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.541960 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1751381870, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.542389 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=46505074, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.542780 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=514284517, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.543225 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1408244156, 16) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.543803 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=480666882, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.544239 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=708474737, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.544672 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1950980479, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.545212 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1129253903, 15) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.545631 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1428286289, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.546054 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1347430492, 13) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.546458 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=22622816, 16) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.546854 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=763956647, 16) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.547260 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1593135939, 14) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.547771 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1063421926, 16) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC W0628 23:54:51.548312 28974 meta_cache.cc:706] Not found: GetTableLocations(table-0, int32 key=1627093773, 16) failed: The table was deleted: Table deleted at 2016-06-28 23:54:51 UTC I0628 23:54:51.549386 27213 external_mini_cluster.cc:333] Process with UUID 7333285db9804dde9478033cabd5d781 has crashed ../../src/kudu/integration-tests/external_mini_cluster.cc:337: Failure Value of: num_crashes Actual: true Expected: 0 At least one process crashed ../../src/kudu/integration-tests/delete_table-test.cc:406: Failure Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread. Actual: it does. I0628 23:54:51.551631 27213 external_mini_cluster-itest-base.h:43] Found fatal failure I0628 23:54:51.551856 27213 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 0 with UUID 420c8e80c83b4919bd67708634ad5bd9 and pid 28602 ************************ BEGIN STACKS ************************** W0628 23:54:51.606721 28982 catalog_manager.cc:1957] TS 7333285db9804dde9478033cabd5d781: Delete Tablet RPC failed for tablet a16b944a988b4388b55f8396b18aa8b9: Network error: Client connection negotiation failed: client connection to 127.106.77.2:40485: connect: Connection refused (error 111) I0628 23:54:51.606878 28982 catalog_manager.cc:2014] Scheduling retry of a16b944a988b4388b55f8396b18aa8b9 Delete Tablet RPC for TS=7333285db9804dde9478033cabd5d781 with a delay of 70ms (attempt = 3)... W0628 23:54:51.679134 28982 catalog_manager.cc:1957] TS 7333285db9804dde9478033cabd5d781: Delete Tablet RPC failed for tablet a16b944a988b4388b55f8396b18aa8b9: Network error: Client connection negotiation failed: client connection to 127.106.77.2:40485: connect: Connection refused (error 111) I0628 23:54:51.679333 28982 catalog_manager.cc:2014] Scheduling retry of a16b944a988b4388b55f8396b18aa8b9 Delete Tablet RPC for TS=7333285db9804dde9478033cabd5d781 with a delay of 153ms (attempt = 4)... 0x00007f396641eb9d in ?? () Id Target Id Frame * 1 process 28602 "kudu-tserver" 0x00007f396641eb9d in ?? () Thread 1 (process 28602): #0 0x00007f396641eb9d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0628 23:54:51.698168 27213 external_mini_cluster-itest-base.h:49] Attempting to dump stacks of TS 1 with UUID a19aaebe5ce646349f024b13675e09ed and pid 28730 ************************ BEGIN STACKS ************************** W0628 23:54:51.834983 28982 catalog_manager.cc:1957] TS 7333285db9804dde9478033cabd5d781: Delete Tablet RPC failed for tablet a16b944a988b4388b55f8396b18aa8b9: Network error: Client connection negotiation failed: client connection to 127.106.77.2:40485: connect: Connection refused (error 111) I0628 23:54:51.835167 28982 catalog_manager.cc:2014] Scheduling retry of a16b944a988b4388b55f8396b18aa8b9 Delete Tablet RPC for TS=7333285db9804dde9478033cabd5d781 with a delay of 267ms (attempt = 5)... 0x00007f2f16225b9d in ?? () Id Target Id Frame * 1 process 28730 "kudu-tserver" 0x00007f2f16225b9d in ?? () Thread 1 (process 28730): #0 0x00007f2f16225b9d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0628 23:54:51.843586 27213 external_mini_cluster-itest-base.h:46] Tablet server 2 is not running. Cannot dump its stacks. I0628 23:54:51.843760 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 28558 I0628 23:54:51.864475 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 28602 I0628 23:54:51.880894 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 28730 I0628 23:54:51.899366 27213 test_util.cc:70] ----------------------------------------------- I0628 23:54:51.899749 27213 test_util.cc:71] Had fatal failures, leaving test files at /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestDeleteTableWithConcurrentWrites.1467158082783852-27213 [ FAILED ] DeleteTableTest.TestDeleteTableWithConcurrentWrites (2858 ms) [ RUN ] DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap I0628 23:54:51.903080 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:52.017992 29072 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:52.018486 29072 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:52.030666 29072 master_main.cc:58] Initializing master server... I0628 23:54:52.032748 29072 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 132351 I0628 23:54:52.039366 29072 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:54:52.039603 29072 server_base.cc:165] Creating new FS layout I0628 23:54:52.052250 29072 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/instance: uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:52.064697 29072 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:52.096750 29072 master_main.cc:61] Starting Master server... I0628 23:54:52.131536 29072 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:53705 I0628 23:54:52.133067 29072 webserver.cc:122] Starting webserver on localhost:0 I0628 23:54:52.133172 29072 webserver.cc:133] Document root disabled I0628 23:54:52.133863 29072 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:54:52.135177 29072 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:41762/ I0628 23:54:52.136471 29072 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:54:52.141424 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 29072 I0628 23:54:52.142321 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:53705 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:52.160923 29115 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Bootstrap starting. I0628 23:54:52.164468 29115 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:54:52.164970 29115 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: No blocks or log segments found. Creating new log. I0628 23:54:52.166503 29115 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:52.170575 29115 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: No bootstrap required, opened a new log I0628 23:54:52.175837 29115 raft_consensus.cc:269] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } I0628 23:54:52.176906 29115 raft_consensus.cc:301] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:52.177139 29115 raft_consensus.cc:499] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c26b7b5a77243a7819c0a4de3bd7a58, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:52.177530 29115 consensus_queue.cc:162] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:52.177778 29115 raft_consensus.cc:314] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:54:52.177908 29115 raft_consensus.cc:370] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:52.178033 29115 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:52.179467 29115 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.649s I0628 23:54:52.179651 29115 raft_consensus.cc:386] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } I0628 23:54:52.181598 29115 leader_election.cc:248] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:52.183089 29120 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.516s I0628 23:54:52.183332 29120 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:52.185588 29120 raft_consensus.cc:464] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 LEADER]: Becoming Leader. State: Replica: 8c26b7b5a77243a7819c0a4de3bd7a58, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:52.186399 29120 consensus_queue.cc:145] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } I0628 23:54:52.205814 29115 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" config { opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } } I0628 23:54:52.206342 29115 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: This master's current role is: LEADER I0628 23:54:52.212106 29122 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c26b7b5a77243a7819c0a4de3bd7a58. Latest consensus state: current_term: 1 leader_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" config { opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } } I0628 23:54:52.212713 29122 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: This master's current role is: LEADER I0628 23:54:52.211688 29121 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" config { opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } } I0628 23:54:52.212971 29121 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: This master's current role is: LEADER I0628 23:54:52.214882 29124 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:54:52.215507 29123 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:54:52.216001 29115 sys_catalog.cc:311] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:54:52.224370 29072 master_main.cc:64] Master server successfully started. I0628 23:54:52.259187 29116 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:52.259678 29116 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:52.284921 29116 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:52.287559 29116 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 132351 I0628 23:54:52.293390 29116 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:54:52.293587 29116 server_base.cc:165] Creating new FS layout I0628 23:54:52.301539 29116 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "9da6f1b0065f4a26bba4b834eae6672b" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:52.310359 29116 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 uuid: "9da6f1b0065f4a26bba4b834eae6672b" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:52.368456 29116 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:52.757936 29116 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:47992 I0628 23:54:52.759409 29116 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:54:52.759534 29116 webserver.cc:133] Document root disabled I0628 23:54:52.761082 29116 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:60859/ I0628 23:54:52.762444 29116 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:54:52.763838 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 29116 I0628 23:54:52.764751 29116 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:52.764844 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:53705 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:52.778656 29240 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53705 I0628 23:54:52.778980 29240 heartbeater.cc:319] Registering TS with master... I0628 23:54:52.779181 29240 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:52.780885 29092 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "9da6f1b0065f4a26bba4b834eae6672b" instance_seqno: 1467158092739631 } with Master I0628 23:54:52.876006 29244 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:52.876482 29244 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:52.889333 29244 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:52.891952 29244 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 132351 I0628 23:54:52.898047 29244 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:54:52.898219 29244 server_base.cc:165] Creating new FS layout I0628 23:54:52.906041 29244 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "61683f288f344bd6a942758e662c2ed9" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:52.913408 29244 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-1 uuid: "61683f288f344bd6a942758e662c2ed9" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:52.945643 29244 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:53.034373 29244 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:34380 I0628 23:54:53.035843 29244 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:54:53.035930 29244 webserver.cc:133] Document root disabled I0628 23:54:53.037415 29244 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:56281/ I0628 23:54:53.038604 29244 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:54:53.040930 29244 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:53.044363 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 29244 I0628 23:54:53.045164 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:53705 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:53.050700 29358 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53705 I0628 23:54:53.051234 29358 heartbeater.cc:319] Registering TS with master... I0628 23:54:53.051481 29358 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:53.052747 29092 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" instance_seqno: 1467158093016915 } with Master I0628 23:54:53.160334 29361 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:53.160915 29361 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:53.175781 29361 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:53.178587 29361 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 132851 I0628 23:54:53.184739 29361 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:54:53.185009 29361 server_base.cc:165] Creating new FS layout I0628 23:54:53.193217 29361 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "11560e5dcebf4497b20737680a0ae704" format_stamp: "Formatted at 2016-06-28 23:54:53 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:53.201565 29361 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-2 uuid: "11560e5dcebf4497b20737680a0ae704" format_stamp: "Formatted at 2016-06-28 23:54:53 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:53.238692 29361 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:53.326308 29361 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:39339 I0628 23:54:53.327707 29361 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:54:53.327786 29361 webserver.cc:133] Document root disabled I0628 23:54:53.329397 29361 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:49514/ I0628 23:54:53.330811 29361 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:54:53.333079 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 29361 I0628 23:54:53.339534 29361 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:53.343978 29475 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53705 I0628 23:54:53.344352 29475 heartbeater.cc:319] Registering TS with master... I0628 23:54:53.344558 29475 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:53.345744 29092 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" instance_seqno: 1467158093309140 } with Master I0628 23:54:53.346771 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:54:53.358079 27213 delete_table-test.cc:419] Shutting down TS 9da6f1b0065f4a26bba4b834eae6672b I0628 23:54:53.358201 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 29116 I0628 23:54:53.373178 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 29072 I0628 23:54:53.390408 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:53705 --webserver_interface=localhost --webserver_port=41762 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:53.499773 29485 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:53.500231 29485 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:53.512106 29485 master_main.cc:58] Initializing master server... I0628 23:54:53.514091 29485 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 132851 I0628 23:54:53.525759 29485 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:53.558513 29485 master_main.cc:61] Starting Master server... I0628 23:54:53.593127 29485 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:53705 I0628 23:54:53.594764 29485 webserver.cc:122] Starting webserver on localhost:41762 I0628 23:54:53.594949 29485 webserver.cc:133] Document root disabled I0628 23:54:53.595731 29485 net_util.cc:183] Address 127.0.0.1:41762 for localhost:41762 duplicates an earlier resolved entry. I0628 23:54:53.597162 29485 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:41762/ I0628 23:54:53.598644 29485 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:54:53.607403 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 29485 I0628 23:54:53.608425 29528 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Bootstrap starting. I0628 23:54:53.613726 29528 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:53.614392 29528 tablet_bootstrap.cc:609] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 I0628 23:54:53.614500 29528 tablet_bootstrap.cc:617] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000 to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery in preparation for log replay W0628 23:54:53.615798 29528 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0628 23:54:53.615990 29528 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:54:53.643872 29528 log_util.cc:569] Scanning /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 for valid entry headers following offset 212... I0628 23:54:53.750005 29528 log_util.cc:606] Found no log entry headers I0628 23:54:53.750720 29528 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 200 for log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001: Prior entries: [off=174 REPLICATE (1.1)] [off=200 COMMIT (1.1)] I0628 23:54:53.750944 29528 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 (valid entries through byte offset 200) I0628 23:54:53.751893 29528 tablet.cc:854] T 00000000000000000000000000000000 Rewinding schema during bootstrap to Schema [ 10:entry_type[int8 NOT NULL], 11:entry_id[string NOT NULL], 12:metadata[string NOT NULL] ] I0628 23:54:53.755239 29528 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:53.761111 29528 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:54:53.761339 29528 tablet_bootstrap.cc:1030] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: ReplayState: Previous OpId: 1.1, Committed OpId: 1.1, Pending Replicates: 0, Pending Commits: 0 I0628 23:54:53.764379 29528 tablet_bootstrap.cc:646] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery I0628 23:54:53.764539 29528 tablet_bootstrap.cc:649] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery-1467158093764496 I0628 23:54:53.764685 29528 tablet_bootstrap.cc:659] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery-1467158093764496 I0628 23:54:53.765375 29528 tablet_bootstrap.cc:662] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery-1467158093764496 I0628 23:54:53.765614 29528 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58: Bootstrap complete. I0628 23:54:53.776465 29528 raft_consensus.cc:269] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } I0628 23:54:53.777549 29528 raft_consensus.cc:499] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c26b7b5a77243a7819c0a4de3bd7a58, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 1 Committed: term: 1 index: 1} I0628 23:54:53.778029 29528 consensus_queue.cc:162] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:53.778321 29528 raft_consensus.cc:314] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:54:53.778467 29528 raft_consensus.cc:370] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:53.778584 29528 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 1 FOLLOWER]: Advancing to term 2 I0628 23:54:53.780134 29528 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.517s I0628 23:54:53.780385 29528 raft_consensus.cc:386] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } I0628 23:54:53.782394 29528 leader_election.cc:248] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0628 23:54:53.783824 29533 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.574s I0628 23:54:53.784050 29533 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 2 FOLLOWER]: Leader election won for term 2 I0628 23:54:53.786103 29533 raft_consensus.cc:464] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 2 LEADER]: Becoming Leader. State: Replica: 8c26b7b5a77243a7819c0a4de3bd7a58, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 1 Committed: term: 1 index: 1} I0628 23:54:53.786736 29533 consensus_queue.cc:145] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 1.1, Committed index: 1.1, Last appended: 1.1, Current term: 2, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } I0628 23:54:53.788573 29528 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 2 leader_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" config { opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } } I0628 23:54:53.788619 29535 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c26b7b5a77243a7819c0a4de3bd7a58. Latest consensus state: current_term: 2 leader_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" config { opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } } I0628 23:54:53.789774 29535 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: This master's current role is: LEADER I0628 23:54:53.789640 29534 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" config { opid_index: -1 peers { permanent_uuid: "8c26b7b5a77243a7819c0a4de3bd7a58" member_type: VOTER } } I0628 23:54:53.790199 29528 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: This master's current role is: LEADER I0628 23:54:53.790596 29534 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: This master's current role is: LEADER I0628 23:54:53.792160 29536 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [term 2 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 1 index: 1 New committed index is: term: 2 index: 2 I0628 23:54:53.794170 29528 sys_catalog.cc:311] T 00000000000000000000000000000000 P 8c26b7b5a77243a7819c0a4de3bd7a58 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:54:53.795783 29537 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:54:53.796375 29485 master_main.cc:64] Master server successfully started. I0628 23:54:54.060211 29505 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" instance_seqno: 1467158093016915 } as {real_user=slave, eff_user=} at 127.106.77.1:55711; Asking this server to re-register. I0628 23:54:54.061461 29358 heartbeater.cc:319] Registering TS with master... I0628 23:54:54.061636 29358 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:54.063091 29505 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" instance_seqno: 1467158093016915 } with Master I0628 23:54:54.351476 29504 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" instance_seqno: 1467158093309140 } as {real_user=slave, eff_user=} at 127.106.77.2:54254; Asking this server to re-register. I0628 23:54:54.352514 29475 heartbeater.cc:319] Registering TS with master... I0628 23:54:54.352694 29475 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:54.353698 29505 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" instance_seqno: 1467158093309140 } with Master I0628 23:54:54.356940 27213 external_mini_cluster.cc:319] 2 TS(s) registered with Master I0628 23:54:54.369200 29505 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:38422: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:54:54.412732 29412 tablet_service.cc:612] Processing CreateTablet for tablet dbdf7f1d38274f1ba048492cfc62819e (table=test-workload [id=c126bf6125a64e70906798331152c17d]), partition=range: [(), ()) I0628 23:54:54.412734 29295 tablet_service.cc:612] Processing CreateTablet for tablet dbdf7f1d38274f1ba048492cfc62819e (table=test-workload [id=c126bf6125a64e70906798331152c17d]), partition=range: [(), ()) I0628 23:54:54.426779 29295 ts_tablet_manager.cc:760] Registered tablet dbdf7f1d38274f1ba048492cfc62819e I0628 23:54:54.426874 29412 ts_tablet_manager.cc:760] Registered tablet dbdf7f1d38274f1ba048492cfc62819e I0628 23:54:54.428406 29551 ts_tablet_manager.cc:628] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9: Bootstrapping tablet I0628 23:54:54.429296 29552 ts_tablet_manager.cc:628] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704: Bootstrapping tablet I0628 23:54:54.430491 29551 tablet_bootstrap.cc:393] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9: Bootstrap starting. I0628 23:54:54.431444 29552 tablet_bootstrap.cc:393] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704: Bootstrap starting. I0628 23:54:54.434201 29551 tablet_bootstrap.cc:553] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:54.434595 29552 tablet_bootstrap.cc:553] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:54.434664 29551 tablet_bootstrap.cc:493] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9: No blocks or log segments found. Creating new log. I0628 23:54:54.435012 29552 tablet_bootstrap.cc:493] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704: No blocks or log segments found. Creating new log. I0628 23:54:54.436347 29551 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:54.436648 29552 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:54.441901 29552 tablet_bootstrap.cc:393] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704: No bootstrap required, opened a new log I0628 23:54:54.442139 29551 tablet_bootstrap.cc:393] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9: No bootstrap required, opened a new log I0628 23:54:54.442227 29552 ts_tablet_manager.cc:633] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s I0628 23:54:54.442560 29551 ts_tablet_manager.cc:633] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s I0628 23:54:54.447693 29552 raft_consensus.cc:269] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } I0628 23:54:54.448599 29551 raft_consensus.cc:269] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } I0628 23:54:54.449398 29552 raft_consensus.cc:301] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:54.449713 29552 raft_consensus.cc:499] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11560e5dcebf4497b20737680a0ae704, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:54.450291 29552 consensus_queue.cc:162] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:54.450306 29551 raft_consensus.cc:301] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:54.450649 29551 raft_consensus.cc:499] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61683f288f344bd6a942758e662c2ed9, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:54.451073 29551 consensus_queue.cc:162] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:54.452775 29552 ts_tablet_manager.cc:659] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s I0628 23:54:54.453790 29551 ts_tablet_manager.cc:659] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s I0628 23:54:54.575234 29559 raft_consensus.cc:370] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:54.575520 29559 raft_consensus.cc:2019] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:54.577795 29559 raft_consensus.cc:1970] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.649s I0628 23:54:54.578043 29559 raft_consensus.cc:386] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } I0628 23:54:54.581254 29559 leader_election.cc:223] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [CANDIDATE]: Term 1 election: Requesting vote from peer 11560e5dcebf4497b20737680a0ae704 I0628 23:54:54.586372 29432 raft_consensus.cc:2019] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:54.588425 29432 raft_consensus.cc:1970] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.536s I0628 23:54:54.589958 29432 raft_consensus.cc:1677] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61683f288f344bd6a942758e662c2ed9 in term 1. I0628 23:54:54.591253 29250 leader_election.cc:347] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [CANDIDATE]: Term 1 election: Vote granted by peer 11560e5dcebf4497b20737680a0ae704 I0628 23:54:54.591522 29250 leader_election.cc:248] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:54.592054 29561 raft_consensus.cc:1970] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.508s I0628 23:54:54.592285 29561 raft_consensus.cc:1859] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:54.596659 29561 raft_consensus.cc:464] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 1 LEADER]: Becoming Leader. State: Replica: 61683f288f344bd6a942758e662c2ed9, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:54.597287 29561 consensus_queue.cc:145] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } I0628 23:54:54.619776 29504 catalog_manager.cc:1673] Tablet: dbdf7f1d38274f1ba048492cfc62819e reported consensus state change. New consensus state: current_term: 1 leader_uuid: "61683f288f344bd6a942758e662c2ed9" config { opid_index: -1 peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } } W0628 23:54:54.620661 29564 raft_consensus_state.cc:541] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:54:54.723443 29432 raft_consensus.cc:869] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [term 1 FOLLOWER]: Refusing update from remote peer 61683f288f344bd6a942758e662c2ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:54:54.724843 29561 consensus_queue.cc:578] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [LEADER]: Connected to new peer: Peer: 11560e5dcebf4497b20737680a0ae704, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:54:54.752562 29564 raft_consensus_state.cc:534] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 2 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==29244==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==29361==WARNING: Can't read from symbolizer at fd 26 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==29361==WARNING: Can't read from symbolizer at fd 26 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==29244==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==29361==WARNING: external symbolizer didn't start up correctly! ==29361==WARNING: Can't write to symbolizer at fd 29 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==29361==WARNING: external symbolizer didn't start up correctly! ==29361==WARNING: Can't write to symbolizer at fd 29 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==29244==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==29361==WARNING: external symbolizer didn't start up correctly! ==29361==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=29361) Write of size 8 at 0x7d700006f828 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f828 by thread T125: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29574, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=29579, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=29361) Atomic write of size 8 at 0x7d700006f820 by thread T123: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f821 by thread T125: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29574, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=29579, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=29361) Write of size 8 at 0x7d700006f8a0 by thread T123: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8a0 by thread T125: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) ==29244==WARNING: Can't read from symbolizer at fd 27 #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29574, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=29579, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=29361) Atomic write of size 4 at 0x7d700006f8f0 by thread T123: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T125: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29574, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=29579, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=29361) Atomic write of size 8 at 0x7d700006f830 by thread T123: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f831 by thread T125: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29574, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=29579, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=29361) Write of size 8 at 0x7d700006f8a8 by thread T123: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8a8 by thread T124: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29574, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=29577, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ==29244==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=29244) Write of size 8 at 0x7d700006f828 by thread T128: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f829 by thread T127: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) ================== WARNING: ThreadSanitizer: data race (pid=29361) Write of size 8 at 0x7d700006f808 by thread T125: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f808 by thread T126: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000002960ab) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000298858) #5 (libtablet.so+0x0000002979a4) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=29579, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=29583, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=29574, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) ================== #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) ================== WARNING: ThreadSanitizer: data race (pid=29361) #17 (libtablet.so+0x0000001ddcaf) Write of size 8 at 0x7d500002fd38 by thread T123: #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #0 (libtablet.so+0x0000001fb20c) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #1 (libtablet.so+0x0000001fae00) #22 (libtablet.so+0x00000023d1ea) #2 (libtablet.so+0x000000295cf2) #23 (libconsensus.so+0x0000000befdd) #3 (libtablet.so+0x00000028c6b7) #24 (libconsensus.so+0x0000000c5726) #4 (libtablet.so+0x000000289219) #25 (libconsensus.so+0x0000000c568e) #5 (libtablet.so+0x0000001dcebd) #26 (libconsensus.so+0x0000000c5643) #6 (libtablet.so+0x0000001dde5d) #27 (libconsensus.so+0x0000000c5401) #7 (libtablet.so+0x0000001ddcaf) #28 (libkrpc.so+0x0000000cce71) #8 (libtablet.so+0x0000002438f1) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #9 (libtablet.so+0x00000023b2f5) #31 (libkudu_util.so+0x00000022941e) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #32 (libkudu_util.so+0x00000022937b) #12 (libtablet.so+0x00000023d1ea) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #13 (libconsensus.so+0x0000000befdd) #35 (libkrpc.so+0x0000000cce71) #14 (libconsensus.so+0x0000000c5726) #36 (libkudu_util.so+0x00000021b55f) #15 (libconsensus.so+0x0000000c568e) #37 (kudu-tserver+0x000000421829) #16 (libconsensus.so+0x0000000c5643) Thread T128 'apply [worker]-' #17 (libconsensus.so+0x0000000c5401) (tid=29575, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #18 (libkrpc.so+0x0000000cce71) #1 (libkudu_util.so+0x00000021ade7) #19 (libkudu_util.so+0x000000226bcd) #2 (libkudu_util.so+0x0000002262ae) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #3 (libkudu_util.so+0x000000224304) #22 (libkudu_util.so+0x00000022937b) #4 (libkudu_util.so+0x000000224902) #23 (libkudu_util.so+0x000000229303) #5 (libkudu_util.so+0x000000224704) #24 (libkudu_util.so+0x000000229101) #6 (libkudu_util.so+0x000000224632) #25 (libkrpc.so+0x0000000cce71) #7 (libtablet.so+0x00000023ae76) #26 (libkudu_util.so+0x00000021b55f) #8 (libtablet.so+0x00000023908d) #27 (kudu-tserver+0x000000421829) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) Previous read of size 8 at 0x7d500002fd38 by thread T126: #11 (libtablet.so+0x00000023444a) #0 (libtablet.so+0x0000001fb19b) #12 (libmaster_rpc.so+0x000000011c31) #1 (libtablet.so+0x0000001fad02) #13 (libconsensus.so+0x000000096280) #2 (libtablet.so+0x000000295cf2) #14 (libconsensus.so+0x0000000f0f2d) #3 (libtablet.so+0x00000028c6b7) #15 (libconsensus.so+0x0000000f0017) #4 (libtablet.so+0x000000289219) #16 (libconsensus.so+0x0000000d76b1) #5 (libtablet.so+0x0000001dcebd) #17 (libconsensus.so+0x0000000d73d0) #6 (libtablet.so+0x0000001dde5d) #18 (libconsensus.so+0x0000000d79cd) #7 (libtablet.so+0x0000001ddcaf) #19 (libconsensus.so+0x0000000a97f2) #8 (libtablet.so+0x0000002438f1) #20 (libconsensus.so+0x0000000ba5e8) #9 (libtablet.so+0x00000023b2f5) #21 (libconsensus.so+0x0000000ba558) #10 (libtablet.so+0x00000023d2c6) #22 (libconsensus.so+0x0000000ba4d3) #11 (libtablet.so+0x00000023d255) #23 (libconsensus.so+0x0000000befdd) #12 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000c5726) #13 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c568e) #14 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c5643) #15 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5401) #16 (libconsensus.so+0x0000000c5643) #28 (libkrpc.so+0x0000000cce71) #17 (libconsensus.so+0x0000000c5401) #29 (libkudu_util.so+0x000000226bcd) #18 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000224fcf) #19 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x00000022941e) #20 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #21 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x000000229101) #22 (libkudu_util.so+0x00000022937b) #35 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x00000021b55f) #24 (libkudu_util.so+0x000000229101) #37 (kudu-tserver+0x000000421829) #25 (libkrpc.so+0x0000000cce71) Thread T127 'apply [worker]-' #26 (libkudu_util.so+0x00000021b55f) (tid=29573, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #1 (libkudu_util.so+0x00000021ade7) #0 (kudu-tserver+0x0000004480c7) #2 (libkudu_util.so+0x0000002262ae) #1 (libtserver.so+0x0000001557a6) #3 (libkudu_util.so+0x000000224304) #2 (libtablet.so+0x0000001fc32a) #4 (libkudu_util.so+0x000000224902) #3 (libtablet.so+0x0000001fb0d9) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #4 (libtablet.so+0x0000001fadd1) #7 (libtablet.so+0x00000023ae76) #5 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000023908d) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #9 (libtablet.so+0x000000234615) #8 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x000000234508) #9 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x00000023444a) #10 (libtablet.so+0x0000001ddcaf) #12 (libmaster_rpc.so+0x000000011c31) #11 (libtablet.so+0x0000002438f1) #13 (libconsensus.so+0x000000096280) #12 (libtablet.so+0x00000023b2f5) #14 (libconsensus.so+0x0000000f0f2d) #13 (libtablet.so+0x00000023d2c6) #15 (libconsensus.so+0x0000000f0017) #14 (libtablet.so+0x00000023d255) #16 (libconsensus.so+0x0000000d76b1) #15 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000d73d0) #16 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000d79cd) #17 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000a97f2) #18 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000ba5e8) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #19 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000ba558) #20 (libconsensus.so+0x0000000c5401) #22 (libconsensus.so+0x0000000ba4d3) #21 (libkrpc.so+0x0000000cce71) #23 (libconsensus.so+0x0000000befdd) #22 (libkudu_util.so+0x000000226bcd) #24 (libconsensus.so+0x0000000c5726) #23 (libkudu_util.so+0x000000224fcf) #25 (libconsensus.so+0x0000000c568e) #24 (libkudu_util.so+0x00000022941e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #25 (libkudu_util.so+0x00000022937b) #28 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000229101) #30 (libkudu_util.so+0x000000224fcf) #28 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000021b55f) #32 (libkudu_util.so+0x00000022937b) #30 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500002fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #33 (libkudu_util.so+0x000000229303) #1 (libtablet.so+0x0000001db43e) #34 (libkudu_util.so+0x000000229101) #2 (libtablet.so+0x000000210662) #35 (libkrpc.so+0x0000000cce71) #3 (libtablet.so+0x00000020f9b7) #36 (libkudu_util.so+0x00000021b55f) #4 (libtablet.so+0x00000020f54e) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' #5 (libtserver.so+0x00000014bc0e) (tid=29572, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #6 (libtserver.so+0x000000157d97) #1 (libkudu_util.so+0x00000021ade7) #7 (libtserver.so+0x000000157ce3) #2 (libkudu_util.so+0x0000002262ae) #8 (libtserver.so+0x000000157c63) #3 (libkudu_util.so+0x000000224304) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #11 (libkudu_util.so+0x000000226bcd) #6 (libkudu_util.so+0x000000224632) #12 (libkudu_util.so+0x000000224fcf) #7 (libtablet.so+0x00000023ae76) #13 (libkudu_util.so+0x00000022941e) #8 (libtablet.so+0x00000023908d) #14 (libkudu_util.so+0x00000022937b) #9 (libtablet.so+0x000000234615) #15 (libkudu_util.so+0x000000229303) #10 (libtablet.so+0x000000234508) #16 (libkudu_util.so+0x000000229101) #11 (libtablet.so+0x00000023444a) #17 (libkrpc.so+0x0000000cce71) #12 (libmaster_rpc.so+0x000000011c31) #18 (libkudu_util.so+0x00000021b55f) #13 (libconsensus.so+0x000000096280) #19 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' #14 (libconsensus.so+0x0000000f0f2d) (tid=29574, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #15 (libconsensus.so+0x0000000f00a1) #1 (libkudu_util.so+0x00000021ade7) #16 (libconsensus.so+0x0000000d76b1) #2 (libkudu_util.so+0x0000002262ae) #17 (libconsensus.so+0x0000000d73d0) #3 (libkudu_util.so+0x000000224304) #18 (libconsensus.so+0x0000000d79cd) #4 (libkudu_util.so+0x000000224902) #19 (libconsensus.so+0x0000000a97f2) #5 (libkudu_util.so+0x000000224704) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #6 (libkudu_util.so+0x000000224632) #22 (libconsensus.so+0x0000000ba4d3) #7 (libtablet.so+0x00000023ae76) #23 (libconsensus.so+0x0000000befdd) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #24 (libconsensus.so+0x0000000c5726) #10 (libtablet.so+0x000000234508) #25 (libconsensus.so+0x0000000c568e) #11 (libtablet.so+0x00000023444a) #26 (libconsensus.so+0x0000000c5643) #12 (libmaster_rpc.so+0x000000011c31) #27 (libconsensus.so+0x0000000c5401) #13 (libconsensus.so+0x000000096280) #28 (libkrpc.so+0x0000000cce71) #14 (libconsensus.so+0x0000000f0f2d) #29 (libkudu_util.so+0x000000226bcd) #15 (libconsensus.so+0x0000000d93bb) #30 (libkudu_util.so+0x000000224fcf) #16 (libconsensus.so+0x0000000d8c4f) #31 (libkudu_util.so+0x00000022941e) #17 (libtserver.so+0x00000012dafc) #32 (libkudu_util.so+0x00000022937b) #18 (libconsensus_proto.so+0x00000006d374) #33 (libkudu_util.so+0x000000229303) #19 (libconsensus_proto.so+0x00000006d18e) #34 (libkudu_util.so+0x000000229101) #20 (libkrpc.so+0x000000121d48) #35 (libkrpc.so+0x0000000cce71) #21 (libkrpc.so+0x000000121948) #36 (libkudu_util.so+0x00000021b55f) #22 (libkrpc.so+0x00000012369d) #37 (kudu-tserver+0x000000421829) #23 (libkrpc.so+0x000000125476) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) ================== #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=29583, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=29552, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=29244) Atomic write of size 8 at 0x7d700006f820 by thread T128: #0 (kudu-tserver+0x000000466ec0) ================== WARNING: ThreadSanitizer: data race (pid=29361) Write of size 8 at 0x7d500002fd38 by thread T125: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #1 (libtablet.so+0x000000227502) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #2 (libtablet.so+0x0000002972a5) #11 (libtablet.so+0x00000023d255) #3 (libtablet.so+0x000000297240) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #4 (libtablet.so+0x00000028bc4a) #14 (libconsensus.so+0x0000000c5726) #5 (libtablet.so+0x000000298a34) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #6 (libtablet.so+0x0000002979d1) #17 (libconsensus.so+0x0000000c5401) #7 (libtablet.so+0x000000296aeb) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #8 (libtablet.so+0x0000002965d3) #20 (libkudu_util.so+0x000000224fcf) #9 (libtablet.so+0x00000028cafd) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #10 (libtablet.so+0x00000028936b) #23 (libkudu_util.so+0x000000229303) #11 (libtablet.so+0x0000001dcebd) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #12 (libtablet.so+0x0000001dde5d) #26 (libkudu_util.so+0x00000021b55f) #13 (libtablet.so+0x0000001ddcaf) #27 (kudu-tserver+0x000000421829) #14 (libtablet.so+0x0000002438f1) Previous read of size 8 at 0x7d500002fd38 by thread T126: #0 (libtablet.so+0x0000001fb19b) #15 (libtablet.so+0x00000023b2f5) #1 (libtablet.so+0x0000001fad02) #16 (libtablet.so+0x00000023d2c6) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #17 (libtablet.so+0x00000023d255) #4 (libtablet.so+0x000000289219) #18 (libtablet.so+0x00000023d1ea) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #19 (libconsensus.so+0x0000000befdd) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #21 (libconsensus.so+0x0000000c568e) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #22 (libconsensus.so+0x0000000c5643) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #23 (libconsensus.so+0x0000000c5401) #2 (libtablet.so+0x0000002960ab) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000298858) #5 (libtablet.so+0x0000002979a4) #24 (libkrpc.so+0x0000000cce71) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #25 (libkudu_util.so+0x000000226bcd) #10 (libtablet.so+0x0000001dcebd) #26 (libkudu_util.so+0x000000224fcf) #11 (libtablet.so+0x0000001dde5d) #27 (libkudu_util.so+0x00000022941e) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #28 (libkudu_util.so+0x00000022937b) #16 (libtablet.so+0x00000023d255) #29 (libkudu_util.so+0x000000229303) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #30 (libkudu_util.so+0x000000229101) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #31 (libkrpc.so+0x0000000cce71) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #24 (libkudu_util.so+0x000000226bcd) #33 (kudu-tserver+0x000000421829) #25 (libkudu_util.so+0x000000224fcf) Previous read of size 1 at 0x7d700006f821 by thread T127 #26 (libkudu_util.so+0x00000022941e) : #0 (libtablet.so+0x0000001fc02e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #1 (libtablet.so+0x0000001fbf59) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #2 (libtablet.so+0x0000001fbbd2) #32 (kudu-tserver+0x000000421829) #3 (libtablet.so+0x0000001fb7ea) Location is heap block of size 456 at 0x7d500002fc00 allocated by thread T117: #4 (libtablet.so+0x0000001fb609) #0 (kudu-tserver+0x0000004aaf73) #5 (libtablet.so+0x0000001fb3e7) #1 (libtablet.so+0x0000001db43e) #6 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000210662) #7 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000020f9b7) #8 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x00000020f54e) #9 (libtablet.so+0x000000289219) #5 (libtserver.so+0x00000014bc0e) #10 (libtablet.so+0x0000001dcebd) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #11 (libtablet.so+0x0000001dde5d) #8 (libtserver.so+0x000000157c63) #12 (libtablet.so+0x0000001ddcaf) #9 (libtserver.so+0x000000157a21) #13 (libtablet.so+0x0000002438f1) #10 (libkrpc.so+0x0000000cce71) #14 (libtablet.so+0x00000023b2f5) #11 (libkudu_util.so+0x000000226bcd) #15 (libtablet.so+0x00000023d2c6) #12 (libkudu_util.so+0x000000224fcf) #16 (libtablet.so+0x00000023d255) #13 (libkudu_util.so+0x00000022941e) #17 (libtablet.so+0x00000023d1ea) #14 (libkudu_util.so+0x00000022937b) #18 (libconsensus.so+0x0000000befdd) #15 (libkudu_util.so+0x000000229303) #19 (libconsensus.so+0x0000000c5726) #16 (libkudu_util.so+0x000000229101) #20 (libconsensus.so+0x0000000c568e) #17 (libkrpc.so+0x0000000cce71) #21 (libconsensus.so+0x0000000c5643) #18 (libkudu_util.so+0x00000021b55f) #22 (libconsensus.so+0x0000000c5401) #19 (kudu-tserver+0x000000421829) #23 (libkrpc.so+0x0000000cce71) Thread T125 #24 (libkudu_util.so+0x000000226bcd) 'apply [worker]-' (tid=29579, running) created by thread T70 at: #25 (libkudu_util.so+0x000000224fcf) #0 (kudu-tserver+0x00000044a123) #26 (libkudu_util.so+0x00000022941e) #1 (libkudu_util.so+0x00000021ade7) #27 (libkudu_util.so+0x00000022937b) #2 (libkudu_util.so+0x0000002262ae) #28 (libkudu_util.so+0x000000229303) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #29 (libkudu_util.so+0x000000229101) #5 (libkudu_util.so+0x000000224704) #30 (libkrpc.so+0x0000000cce71) #6 (libkudu_util.so+0x000000224632) #31 (libkudu_util.so+0x00000021b55f) #7 (libtablet.so+0x00000023ae76) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T126: #8 (libtablet.so+0x00000023908d) #0 (kudu-tserver+0x000000440858) #9 (libtablet.so+0x000000234615) #1 (libkudu_util.so+0x0000001d048f) #10 (libtablet.so+0x000000234508) #2 (libkudu_util.so+0x0000001d039a) #11 (libtablet.so+0x00000023444a) #3 (libkudu_util.so+0x0000001d11a9) #12 (libmaster_rpc.so+0x000000011c31) #4 (libkudu_util.so+0x0000001d0e5f) #13 (libconsensus.so+0x000000096280) #5 (libkudu_util.so+0x0000001cc3ab) #14 (libconsensus.so+0x0000000f0f2d) #6 (libkudu_util.so+0x0000001cb1d4) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #7 (libkudu_util.so+0x0000001cba25) #17 (libtserver.so+0x00000012dafc) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #18 (libconsensus_proto.so+0x00000006d374) #10 (libtablet.so+0x000000297af3) #19 (libconsensus_proto.so+0x00000006d18e) #11 (libtablet.so+0x000000296aeb) #20 (libkrpc.so+0x000000121d48) #12 (libtablet.so+0x0000002965d3) #21 (libkrpc.so+0x000000121948) #13 (libtablet.so+0x00000028cafd) #22 (libkrpc.so+0x00000012369d) #14 (libtablet.so+0x00000028936b) #23 (libkrpc.so+0x000000125476) #15 (libtablet.so+0x0000001dcebd) #24 (libkrpc.so+0x0000001253da) #16 (libtablet.so+0x0000001dde5d) #25 (libkrpc.so+0x000000125383) #17 (libtablet.so+0x0000001ddcaf) #26 (libkrpc.so+0x0000001251a9) #18 (libtablet.so+0x0000002438f1) #27 (libkrpc.so+0x0000000cce71) #19 (libtablet.so+0x00000023b2f5) #28 (libkudu_util.so+0x00000021b55f) #20 (libtablet.so+0x00000023d2c6) #29 (kudu-tserver+0x000000421829) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) Thread T126 'apply [worker]-' (tid=29583, running) created by thread T69 at: #23 (libconsensus.so+0x0000000befdd) #0 (kudu-tserver+0x00000044a123) #24 (libconsensus.so+0x0000000c5726) #1 (libkudu_util.so+0x00000021ade7) #25 (libconsensus.so+0x0000000c568e) #2 (libkudu_util.so+0x0000002262ae) #26 (libconsensus.so+0x0000000c5643) #3 (libkudu_util.so+0x000000224304) #27 (libconsensus.so+0x0000000c5401) #4 (libkudu_util.so+0x000000224902) #28 (libkrpc.so+0x0000000cce71) #5 (libkudu_util.so+0x000000224704) #29 (libkudu_util.so+0x000000226bcd) #6 (libkudu_util.so+0x000000224632) #30 (libkudu_util.so+0x000000224fcf) #7 (libtablet.so+0x00000023ae76) #31 (libkudu_util.so+0x00000022941e) #8 (libtablet.so+0x00000023908d) #32 (libkudu_util.so+0x00000022937b) #9 (libtablet.so+0x000000234615) #33 (libkudu_util.so+0x000000229303) #10 (libtablet.so+0x000000234508) #34 (libkudu_util.so+0x000000229101) #11 (libtablet.so+0x00000023444a) #35 (libkrpc.so+0x0000000cce71) #12 (libmaster_rpc.so+0x000000011c31) #36 (libkudu_util.so+0x00000021b55f) #13 (libconsensus.so+0x000000096280) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' #14 (libconsensus.so+0x0000000f0f2d) (tid=29575, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #15 (libconsensus.so+0x0000000d93bb) #1 (libkudu_util.so+0x00000021ade7) #16 (libconsensus.so+0x0000000d8c4f) #2 (libkudu_util.so+0x0000002262ae) #17 (libtserver.so+0x00000012dafc) #3 (libkudu_util.so+0x000000224304) #18 (libconsensus_proto.so+0x00000006d374) #4 (libkudu_util.so+0x000000224902) #19 (libconsensus_proto.so+0x00000006d18e) #5 (libkudu_util.so+0x000000224704) #20 (libkrpc.so+0x000000121d48) #6 (libkudu_util.so+0x000000224632) #21 (libkrpc.so+0x000000121948) #7 (libtablet.so+0x00000023ae76) #22 (libkrpc.so+0x00000012369d) #8 (libtablet.so+0x00000023908d) #23 (libkrpc.so+0x000000125476) #9 (libtablet.so+0x000000234615) #24 (libkrpc.so+0x0000001253da) #10 (libtablet.so+0x000000234508) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #11 (libtablet.so+0x00000023444a) #27 (libkrpc.so+0x0000000cce71) #12 (libmaster_rpc.so+0x000000011c31) #28 (libkudu_util.so+0x00000021b55f) #13 (libconsensus.so+0x000000096280) #29 (kudu-tserver+0x000000421829) #14 (libconsensus.so+0x0000000f0f2d) Thread T117 'tablet-bootstra' #15 (libconsensus.so+0x0000000f0017) (tid=29552, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #16 (libconsensus.so+0x0000000d76b1) #1 (libkudu_util.so+0x00000021ade7) #17 (libconsensus.so+0x0000000d73d0) #2 (libkudu_util.so+0x0000002262ae) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #3 (libkudu_util.so+0x000000224304) #20 (libconsensus.so+0x0000000ba5e8) #4 (libkudu_util.so+0x000000224902) #21 (libconsensus.so+0x0000000ba558) #5 (libkudu_util.so+0x000000224704) #22 (libconsensus.so+0x0000000ba4d3) #6 (libtserver.so+0x00000014cd50) #23 (libconsensus.so+0x0000000befdd) #7 (libtserver.so+0x00000012b940) #24 (libconsensus.so+0x0000000c5726) #8 (libtserver_admin_proto.so+0x000000028a24) #25 (libconsensus.so+0x0000000c568e) #9 (libtserver_admin_proto.so+0x00000002883e) #26 (libconsensus.so+0x0000000c5643) #10 (libkrpc.so+0x000000121d48) #27 (libconsensus.so+0x0000000c5401) #11 (libkrpc.so+0x000000121948) #28 (libkrpc.so+0x0000000cce71) #12 (libkrpc.so+0x00000012369d) #29 (libkudu_util.so+0x000000226bcd) #13 (libkrpc.so+0x000000125476) #30 (libkudu_util.so+0x000000224fcf) #14 (libkrpc.so+0x0000001253da) #31 (libkudu_util.so+0x00000022941e) #15 (libkrpc.so+0x000000125383) #32 (libkudu_util.so+0x00000022937b) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x000000229303) #18 (libkudu_util.so+0x00000021b55f) #34 (libkudu_util.so+0x000000229101) #19 (kudu-tserver+0x000000421829) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) Thread T127 'apply [worker]-' (tid=29573, running) created by thread T124 at: ================== #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=29572, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=29244) Write of size 8 at 0x7d700006f898 by thread T128: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f898 by thread T127: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29575, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29573, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=29572, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=29244) Atomic write of size 4 at 0x7d700006f8f0 by thread T128: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T127: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29575, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29573, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=29572, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=29244) Atomic write of size 8 at 0x7d700006f830 by thread T127: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f831 by thread T128: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29573, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29575, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=29572, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=29361) Write of size 8 at 0x7d500002fd38 by thread T126: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fae00) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500002fd38 by thread T125: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500002fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=29583, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=29579, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=29552, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=29244) Write of size 8 at 0x7d500002fd38 by thread T128: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500002fd38 by thread T127: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500002fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29575, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29573, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=29551, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=29244) Write of size 8 at 0x7d500002fd38 by thread T127: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500002fd38 by thread T128: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500002fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29573, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=29575, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=29551, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=29244) Read of size 8 at 0x7d500002fd38 by thread T126: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500002fd38 by thread T127: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500002fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=29572, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=29573, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=29551, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== I0628 23:54:54.949558 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:47992 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=60859 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:53705 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:55.057148 29590 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:55.057554 29590 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:55.069766 29590 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:55.072307 29590 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 133851 I0628 23:54:55.083001 29590 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 uuid: "9da6f1b0065f4a26bba4b834eae6672b" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:55.117734 29590 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:55.204516 29590 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:47992 I0628 23:54:55.206116 29590 webserver.cc:122] Starting webserver on 127.106.77.0:60859 I0628 23:54:55.206230 29590 webserver.cc:133] Document root disabled I0628 23:54:55.207867 29590 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:60859/ I0628 23:54:55.209417 29590 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:54:55.211609 29590 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:55.217279 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 29590 I0628 23:54:55.221109 29704 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53705 I0628 23:54:55.221485 29704 heartbeater.cc:319] Registering TS with master... I0628 23:54:55.221709 29704 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:55.222965 29501 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "9da6f1b0065f4a26bba4b834eae6672b" instance_seqno: 1467158095186614 } with Master W0628 23:54:55.225114 29701 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:45510 forcing change of non-runtime-safe flag fault_crash_after_rb_files_fetched I0628 23:54:55.225344 29701 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:45510 changed flags via RPC: fault_crash_after_rb_files_fetched from '0' to '1.0' I0628 23:54:55.235255 29315 consensus_peers.cc:351] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 -> Peer 11560e5dcebf4497b20737680a0ae704 (127.106.77.2:39339): Closing peer: 11560e5dcebf4497b20737680a0ae704 I0628 23:54:55.235843 29315 consensus_queue.cc:145] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [LEADER]: Queue going to LEADER mode. State: All replicated op: 1.7, Majority replicated op: 1.7, Committed index: 1.7, Last appended: 1.7, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } peers { permanent_uuid: "9da6f1b0065f4a26bba4b834eae6672b" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47992 } } I0628 23:54:55.241858 29431 raft_consensus.cc:869] T dbdf7f1d38274f1ba048492cfc62819e P 11560e5dcebf4497b20737680a0ae704 [term 1 FOLLOWER]: Refusing update from remote peer 61683f288f344bd6a942758e662c2ed9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I0628 23:54:55.242980 29561 consensus_queue.cc:578] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [LEADER]: Connected to new peer: Peer: 11560e5dcebf4497b20737680a0ae704, Is new: false, Last received: 1.7, Next index: 8, Last known committed idx: 7, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:54:55.245815 29562 consensus_peers.cc:296] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 -> Peer 9da6f1b0065f4a26bba4b834eae6672b (127.106.77.0:47992): Sending request to remotely bootstrap I0628 23:54:55.247280 29661 ts_tablet_manager.cc:412] T dbdf7f1d38274f1ba048492cfc62819e P 9da6f1b0065f4a26bba4b834eae6672b: Initiating remote bootstrap from Peer 61683f288f344bd6a942758e662c2ed9 (127.106.77.1:34380) I0628 23:54:55.247648 29661 remote_bootstrap_client.cc:164] T dbdf7f1d38274f1ba048492cfc62819e P 9da6f1b0065f4a26bba4b834eae6672b: Remote bootstrap client: Beginning remote bootstrap session from remote peer at address 127.106.77.1:34380 I0628 23:54:55.248411 29564 raft_consensus_state.cc:609] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9 [term 1 LEADER]: Committing config change with OpId 1.8. Old config: { opid_index: -1 peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } }. New config: { opid_index: 8 peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } peers { permanent_uuid: "9da6f1b0065f4a26bba4b834eae6672b" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47992 } } } I0628 23:54:55.254820 29335 remote_bootstrap_service.cc:123] Beginning new remote bootstrap session on tablet dbdf7f1d38274f1ba048492cfc62819e from peer 9da6f1b0065f4a26bba4b834eae6672b at {real_user=slave, eff_user=} at 127.106.77.0:46825: session id = 9da6f1b0065f4a26bba4b834eae6672b-dbdf7f1d38274f1ba048492cfc62819e I0628 23:54:55.255103 29501 catalog_manager.cc:1673] Tablet: dbdf7f1d38274f1ba048492cfc62819e reported consensus state change. New consensus state: current_term: 1 leader_uuid: "61683f288f344bd6a942758e662c2ed9" config { opid_index: 8 peers { permanent_uuid: "11560e5dcebf4497b20737680a0ae704" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 39339 } } peers { permanent_uuid: "61683f288f344bd6a942758e662c2ed9" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 34380 } } peers { permanent_uuid: "9da6f1b0065f4a26bba4b834eae6672b" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 47992 } } } I0628 23:54:55.257928 29335 remote_bootstrap_session.cc:142] T dbdf7f1d38274f1ba048492cfc62819e P 61683f288f344bd6a942758e662c2ed9: Remote bootstrap: opened 0 blocks and 1 log segments I0628 23:54:55.274907 29661 ts_tablet_manager.cc:760] Registered tablet dbdf7f1d38274f1ba048492cfc62819e I0628 23:54:55.275352 29661 remote_bootstrap_client.cc:387] T dbdf7f1d38274f1ba048492cfc62819e P 9da6f1b0065f4a26bba4b834eae6672b: Remote bootstrap client: Starting download of 0 data blocks... I0628 23:54:55.275634 29661 remote_bootstrap_client.cc:352] T dbdf7f1d38274f1ba048492cfc62819e P 9da6f1b0065f4a26bba4b834eae6672b: Remote bootstrap client: Starting download of 1 WAL segments... I0628 23:54:55.275787 29661 tablet_bootstrap.cc:393] T dbdf7f1d38274f1ba048492cfc62819e P 9da6f1b0065f4a26bba4b834eae6672b: RemoteBootstrap: Downloading WAL segment with seq. number 1 (1/1) W0628 23:54:55.278609 29661 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:54:55.278697 29661 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:54:55.278825 29661 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_rb_files_fetched *** Check failure stack trace: *** @ 0x7f3b22f02304 google::LogMessage::SendToLog() at ??:0 @ 0x7f3b22f02800 google::LogMessage::Flush() at ??:0 @ 0x7f3b22f06432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f3b23bcde42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7f3b2aa89e6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f3b2aad0b69 kudu::tserver::TSTabletManager::StartRemoteBootstrap() at ??:0 @ 0x7f3b2aab5360 kudu::tserver::ConsensusServiceImpl::StartRemoteBootstrap() at ??:0 @ 0x7f3b288ec9f5 kudu::consensus::ConsensusServiceIf::ConsensusServiceIf()::$_8::operator()() at ??:0 @ 0x7f3b288ec80f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f3b2819ed49 std::function<>::operator()() at ??:0 @ 0x7f3b2819e949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7f3b281a069e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f3b281a2477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f3b281a23db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f3b281a2384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f3b281a21aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f3b28149e72 boost::function0<>::operator()() at ??:0 @ 0x7f3b23c38560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7f3b25677182 start_thread at ??:0 @ 0x7f3b2120647d clone at ??:0 @ (nil) (unknown) I0628 23:54:55.407421 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 29244 I0628 23:54:55.424828 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 29361 I0628 23:54:55.440815 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:47992 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=60859 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:53705 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:55.554997 29734 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:55.555480 29734 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:55.569499 29734 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:55.572376 29734 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 133851 I0628 23:54:55.584152 29734 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0 uuid: "9da6f1b0065f4a26bba4b834eae6672b" format_stamp: "Formatted at 2016-06-28 23:54:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:55.621062 29734 ts_tablet_manager.cc:604] Loading metadata for tablet dbdf7f1d38274f1ba048492cfc62819e I0628 23:54:55.626381 29734 ts_tablet_manager.cc:936] T dbdf7f1d38274f1ba048492cfc62819e P 9da6f1b0065f4a26bba4b834eae6672b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_TOMBSTONED I0628 23:54:55.632462 29734 ts_tablet_manager.cc:955] T dbdf7f1d38274f1ba048492cfc62819e P 9da6f1b0065f4a26bba4b834eae6672b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:54:55.638361 29734 ts_tablet_manager.cc:965] T dbdf7f1d38274f1ba048492cfc62819e P 9da6f1b0065f4a26bba4b834eae6672b: Tablet deleted. Last logged OpId: 0.0 I0628 23:54:55.639364 29734 ts_tablet_manager.cc:760] Registered tablet dbdf7f1d38274f1ba048492cfc62819e I0628 23:54:55.640365 29734 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:55.728694 29734 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:47992 I0628 23:54:55.730043 29734 webserver.cc:122] Starting webserver on 127.106.77.0:60859 I0628 23:54:55.730149 29734 webserver.cc:133] Document root disabled I0628 23:54:55.731633 29734 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:60859/ I0628 23:54:55.733197 29734 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:54:55.735605 29734 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:55.738420 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 29734 I0628 23:54:55.744451 29848 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53705 I0628 23:54:55.744771 29848 heartbeater.cc:319] Registering TS with master... I0628 23:54:55.744968 29848 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:55.745110 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 29485 I0628 23:54:55.762595 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 29734 [ OK ] DeleteTableTest.TestAutoTombstoneAfterCrashDuringRemoteBootstrap (3881 ms) [ RUN ] DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails I0628 23:54:55.783756 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master --catalog_manager_wait_for_new_tablets_to_elect_leader=false --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:55.897367 29853 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:55.897781 29853 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:55.909947 29853 master_main.cc:58] Initializing master server... I0628 23:54:55.912590 29853 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 133851 I0628 23:54:55.919448 29853 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:54:55.919664 29853 server_base.cc:165] Creating new FS layout I0628 23:54:55.933547 29853 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/instance: uuid: "02903583b1a046dd8d37b22ec88ce12f" format_stamp: "Formatted at 2016-06-28 23:54:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:55.946472 29853 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master uuid: "02903583b1a046dd8d37b22ec88ce12f" format_stamp: "Formatted at 2016-06-28 23:54:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:55.978449 29853 master_main.cc:61] Starting Master server... I0628 23:54:56.012811 29853 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:48620 I0628 23:54:56.014219 29853 webserver.cc:122] Starting webserver on localhost:0 I0628 23:54:56.014313 29853 webserver.cc:133] Document root disabled I0628 23:54:56.015012 29853 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:54:56.016204 29853 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:60679/ I0628 23:54:56.017565 29853 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:54:56.020045 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 29853 I0628 23:54:56.020944 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:48620 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0 --enable_leader_failure_detection=false --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:56.041784 29896 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Bootstrap starting. I0628 23:54:56.045549 29896 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:56.046033 29896 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: No blocks or log segments found. Creating new log. I0628 23:54:56.047652 29896 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:56.051695 29896 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: No bootstrap required, opened a new log I0628 23:54:56.057289 29896 raft_consensus.cc:269] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } I0628 23:54:56.058387 29896 raft_consensus.cc:301] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:54:56.058639 29896 raft_consensus.cc:499] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02903583b1a046dd8d37b22ec88ce12f, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:56.058975 29896 consensus_queue.cc:162] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:56.059267 29896 raft_consensus.cc:314] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:54:56.059434 29896 raft_consensus.cc:370] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:56.059576 29896 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:56.061125 29896 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.567s I0628 23:54:56.061298 29896 raft_consensus.cc:386] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } I0628 23:54:56.063222 29896 leader_election.cc:248] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:56.064723 29901 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.551s I0628 23:54:56.064972 29901 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:56.077611 29901 raft_consensus.cc:464] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 LEADER]: Becoming Leader. State: Replica: 02903583b1a046dd8d37b22ec88ce12f, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:56.078148 29901 consensus_queue.cc:145] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } I0628 23:54:56.096734 29902 sys_catalog.cc:235] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02903583b1a046dd8d37b22ec88ce12f" config { opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } } I0628 23:54:56.097286 29902 sys_catalog.cc:238] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: This master's current role is: LEADER I0628 23:54:56.097200 29896 sys_catalog.cc:235] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "02903583b1a046dd8d37b22ec88ce12f" config { opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } } I0628 23:54:56.097537 29896 sys_catalog.cc:238] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: This master's current role is: LEADER I0628 23:54:56.098301 29903 sys_catalog.cc:235] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02903583b1a046dd8d37b22ec88ce12f. Latest consensus state: current_term: 1 leader_uuid: "02903583b1a046dd8d37b22ec88ce12f" config { opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } } I0628 23:54:56.098870 29903 sys_catalog.cc:238] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: This master's current role is: LEADER I0628 23:54:56.100709 29905 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:54:56.101279 29896 sys_catalog.cc:311] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: configured and running, proceeding with master startup. I0628 23:54:56.106662 29906 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:54:56.120739 29853 master_main.cc:64] Master server successfully started. I0628 23:54:56.134068 29897 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:56.134508 29897 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:56.147557 29897 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:56.150279 29897 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 134351 I0628 23:54:56.156029 29897 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:54:56.156309 29897 server_base.cc:165] Creating new FS layout I0628 23:54:56.164561 29897 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "a9e0262858964a4387d95eaa285fa762" format_stamp: "Formatted at 2016-06-28 23:54:56 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:56.172754 29897 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0 uuid: "a9e0262858964a4387d95eaa285fa762" format_stamp: "Formatted at 2016-06-28 23:54:56 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:56.216550 29897 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:56.631188 29897 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:35960 I0628 23:54:56.632611 29897 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:54:56.632699 29897 webserver.cc:133] Document root disabled I0628 23:54:56.634357 29897 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:37992/ I0628 23:54:56.635733 29897 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:54:56.641973 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 29897 I0628 23:54:56.642695 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:48620 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1 --enable_leader_failure_detection=false --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:56.654366 29897 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:56.664535 30021 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:48620 I0628 23:54:56.664875 30021 heartbeater.cc:319] Registering TS with master... I0628 23:54:56.665103 30021 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:56.666774 29873 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" instance_seqno: 1467158096580711 } with Master I0628 23:54:56.758994 30025 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:56.759451 30025 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:56.773027 30025 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:56.775964 30025 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 134351 I0628 23:54:56.782064 30025 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:54:56.782239 30025 server_base.cc:165] Creating new FS layout I0628 23:54:56.790500 30025 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "8bea06ed38f24dd3b85c977539f7d5f5" format_stamp: "Formatted at 2016-06-28 23:54:56 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:56.798630 30025 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1 uuid: "8bea06ed38f24dd3b85c977539f7d5f5" format_stamp: "Formatted at 2016-06-28 23:54:56 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:56.830310 30025 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:56.920050 30025 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:40602 I0628 23:54:56.921439 30025 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:54:56.921536 30025 webserver.cc:133] Document root disabled I0628 23:54:56.922991 30025 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:48507/ I0628 23:54:56.924381 30025 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:54:56.926781 30025 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:56.929807 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 30025 I0628 23:54:56.930733 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:48620 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2 --enable_leader_failure_detection=false --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:56.936046 30139 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:48620 I0628 23:54:56.936472 30139 heartbeater.cc:319] Registering TS with master... I0628 23:54:56.936697 30139 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:56.937997 29873 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" instance_seqno: 1467158096901961 } with Master I0628 23:54:57.044536 30142 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:57.045001 30142 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:57.057672 30142 tablet_server_main.cc:54] Initializing tablet server... I0628 23:54:57.060688 30142 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 134851 I0628 23:54:57.066315 30142 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:54:57.066521 30142 server_base.cc:165] Creating new FS layout I0628 23:54:57.074417 30142 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "c3a57434d605412c82946ec0f6f80b2a" format_stamp: "Formatted at 2016-06-28 23:54:57 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:57.082249 30142 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2 uuid: "c3a57434d605412c82946ec0f6f80b2a" format_stamp: "Formatted at 2016-06-28 23:54:57 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:57.116616 30142 tablet_server_main.cc:57] Starting tablet server... I0628 23:54:57.209504 30142 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:48322 I0628 23:54:57.210965 30142 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:54:57.211052 30142 webserver.cc:133] Document root disabled I0628 23:54:57.212604 30142 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:46741/ I0628 23:54:57.214004 30142 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:54:57.217362 30142 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:54:57.220818 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 30142 I0628 23:54:57.230345 30256 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:48620 I0628 23:54:57.230726 30256 heartbeater.cc:319] Registering TS with master... I0628 23:54:57.230988 30256 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:57.232270 29873 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" instance_seqno: 1467158097189824 } with Master I0628 23:54:57.233808 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:54:57.245990 27213 delete_table-test.cc:490] Shutting down TS a9e0262858964a4387d95eaa285fa762 I0628 23:54:57.246098 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 29897 I0628 23:54:57.260195 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 29853 I0628 23:54:57.274595 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:48620 --webserver_interface=localhost --webserver_port=60679 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master --catalog_manager_wait_for_new_tablets_to_elect_leader=false --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:54:57.384709 30266 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:54:57.385179 30266 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:54:57.396875 30266 master_main.cc:58] Initializing master server... I0628 23:54:57.398869 30266 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 134851 I0628 23:54:57.411350 30266 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master uuid: "02903583b1a046dd8d37b22ec88ce12f" format_stamp: "Formatted at 2016-06-28 23:54:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:54:57.444732 30266 master_main.cc:61] Starting Master server... I0628 23:54:57.479607 30266 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:48620 I0628 23:54:57.481024 30266 webserver.cc:122] Starting webserver on localhost:60679 I0628 23:54:57.481117 30266 webserver.cc:133] Document root disabled I0628 23:54:57.481796 30266 net_util.cc:183] Address 127.0.0.1:60679 for localhost:60679 duplicates an earlier resolved entry. I0628 23:54:57.483079 30266 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:60679/ I0628 23:54:57.484688 30266 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:54:57.490855 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 30266 I0628 23:54:57.496963 30309 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Bootstrap starting. I0628 23:54:57.502611 30309 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:57.503408 30309 tablet_bootstrap.cc:609] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 I0628 23:54:57.503553 30309 tablet_bootstrap.cc:617] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000 to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery in preparation for log replay W0628 23:54:57.505378 30309 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0628 23:54:57.505568 30309 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:54:57.531191 30309 log_util.cc:569] Scanning /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 for valid entry headers following offset 212... I0628 23:54:57.636723 30309 log_util.cc:606] Found no log entry headers I0628 23:54:57.637547 30309 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 200 for log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001: Prior entries: [off=174 REPLICATE (1.1)] [off=200 COMMIT (1.1)] I0628 23:54:57.637784 30309 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery/wal-000000001 (valid entries through byte offset 200) I0628 23:54:57.638670 30309 tablet.cc:854] T 00000000000000000000000000000000 Rewinding schema during bootstrap to Schema [ 10:entry_type[int8 NOT NULL], 11:entry_id[string NOT NULL], 12:metadata[string NOT NULL] ] I0628 23:54:57.642037 30309 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:57.647837 30309 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:54:57.648067 30309 tablet_bootstrap.cc:1030] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: ReplayState: Previous OpId: 1.1, Committed OpId: 1.1, Pending Replicates: 0, Pending Commits: 0 I0628 23:54:57.651368 30309 tablet_bootstrap.cc:646] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery I0628 23:54:57.651497 30309 tablet_bootstrap.cc:649] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery-1467158097651459 I0628 23:54:57.651638 30309 tablet_bootstrap.cc:659] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery-1467158097651459 I0628 23:54:57.652231 30309 tablet_bootstrap.cc:662] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/master/wals/00000000000000000000000000000000.recovery-1467158097651459 I0628 23:54:57.652454 30309 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f: Bootstrap complete. I0628 23:54:57.662832 30309 raft_consensus.cc:269] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } I0628 23:54:57.663903 30309 raft_consensus.cc:499] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02903583b1a046dd8d37b22ec88ce12f, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 1 Committed: term: 1 index: 1} I0628 23:54:57.664342 30309 consensus_queue.cc:162] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:57.664600 30309 raft_consensus.cc:314] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:54:57.664736 30309 raft_consensus.cc:370] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:57.664872 30309 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 1 FOLLOWER]: Advancing to term 2 I0628 23:54:57.666472 30309 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.606s I0628 23:54:57.666791 30309 raft_consensus.cc:386] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 2 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } I0628 23:54:57.668701 30309 leader_election.cc:248] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0628 23:54:57.670063 30314 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.605s I0628 23:54:57.670433 30314 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 2 FOLLOWER]: Leader election won for term 2 I0628 23:54:57.672565 30314 raft_consensus.cc:464] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 2 LEADER]: Becoming Leader. State: Replica: 02903583b1a046dd8d37b22ec88ce12f, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 1 Committed: term: 1 index: 1} I0628 23:54:57.673434 30314 consensus_queue.cc:145] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 1.1, Committed index: 1.1, Last appended: 1.1, Current term: 2, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } I0628 23:54:57.675658 30309 sys_catalog.cc:235] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 2 leader_uuid: "02903583b1a046dd8d37b22ec88ce12f" config { opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } } I0628 23:54:57.676074 30309 sys_catalog.cc:238] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: This master's current role is: LEADER I0628 23:54:57.677079 30315 sys_catalog.cc:235] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "02903583b1a046dd8d37b22ec88ce12f" config { opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } } I0628 23:54:57.676951 30316 sys_catalog.cc:235] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02903583b1a046dd8d37b22ec88ce12f. Latest consensus state: current_term: 2 leader_uuid: "02903583b1a046dd8d37b22ec88ce12f" config { opid_index: -1 peers { permanent_uuid: "02903583b1a046dd8d37b22ec88ce12f" member_type: VOTER } } I0628 23:54:57.678791 30315 sys_catalog.cc:238] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: This master's current role is: LEADER I0628 23:54:57.678858 30316 sys_catalog.cc:238] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: This master's current role is: LEADER I0628 23:54:57.679272 30317 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [term 2 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 1 index: 1 New committed index is: term: 2 index: 2 I0628 23:54:57.680223 30318 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:54:57.681213 30309 sys_catalog.cc:311] T 00000000000000000000000000000000 P 02903583b1a046dd8d37b22ec88ce12f [sys.catalog]: configured and running, proceeding with master startup. I0628 23:54:57.692894 30266 master_main.cc:64] Master server successfully started. I0628 23:54:57.946158 30286 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" instance_seqno: 1467158096901961 } as {real_user=slave, eff_user=} at 127.106.77.1:58088; Asking this server to re-register. I0628 23:54:57.947264 30139 heartbeater.cc:319] Registering TS with master... I0628 23:54:57.947417 30139 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:57.948885 30286 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" instance_seqno: 1467158096901961 } with Master I0628 23:54:58.238018 30286 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" instance_seqno: 1467158097189824 } as {real_user=slave, eff_user=} at 127.106.77.2:49882; Asking this server to re-register. I0628 23:54:58.239220 30256 heartbeater.cc:319] Registering TS with master... I0628 23:54:58.239457 30256 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:54:58.240733 30286 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" instance_seqno: 1467158097189824 } with Master I0628 23:54:58.242463 27213 external_mini_cluster.cc:319] 2 TS(s) registered with Master I0628 23:54:58.255050 30286 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:49883: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:54:58.298899 30193 tablet_service.cc:612] Processing CreateTablet for tablet 81bb583ab6264dd3bed571d157bdc78e (table=test-workload [id=80dde25440d24e9ca8b305595e464336]), partition=range: [(), ()) I0628 23:54:58.300338 30076 tablet_service.cc:612] Processing CreateTablet for tablet 81bb583ab6264dd3bed571d157bdc78e (table=test-workload [id=80dde25440d24e9ca8b305595e464336]), partition=range: [(), ()) I0628 23:54:58.314657 30193 ts_tablet_manager.cc:760] Registered tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:54:58.315201 30076 ts_tablet_manager.cc:760] Registered tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:54:58.316615 30333 ts_tablet_manager.cc:628] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Bootstrapping tablet I0628 23:54:58.318961 30332 ts_tablet_manager.cc:628] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Bootstrapping tablet I0628 23:54:58.319480 30333 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Bootstrap starting. I0628 23:54:58.321077 30332 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Bootstrap starting. I0628 23:54:58.323027 30333 tablet_bootstrap.cc:553] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:54:58.323570 30333 tablet_bootstrap.cc:493] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: No blocks or log segments found. Creating new log. I0628 23:54:58.324513 30332 tablet_bootstrap.cc:553] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:54:58.324995 30332 tablet_bootstrap.cc:493] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: No blocks or log segments found. Creating new log. I0628 23:54:58.325279 30333 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:58.326622 30332 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:54:58.329875 30333 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: No bootstrap required, opened a new log I0628 23:54:58.330211 30333 ts_tablet_manager.cc:633] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s I0628 23:54:58.330857 30332 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: No bootstrap required, opened a new log I0628 23:54:58.331285 30332 ts_tablet_manager.cc:633] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s I0628 23:54:58.335188 30333 raft_consensus.cc:269] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } I0628 23:54:58.336387 30333 raft_consensus.cc:499] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bea06ed38f24dd3b85c977539f7d5f5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:58.335800 30332 raft_consensus.cc:269] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } I0628 23:54:58.336755 30333 consensus_queue.cc:162] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:58.337092 30332 raft_consensus.cc:499] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a57434d605412c82946ec0f6f80b2a, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:58.337616 30332 consensus_queue.cc:162] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:54:58.339479 30333 ts_tablet_manager.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s I0628 23:54:58.342010 30332 ts_tablet_manager.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s I0628 23:54:58.371009 30096 raft_consensus.cc:370] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:54:58.371397 30096 raft_consensus.cc:2019] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:58.373440 30096 raft_consensus.cc:386] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } I0628 23:54:58.376623 30096 leader_election.cc:223] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [CANDIDATE]: Term 1 election: Requesting vote from peer c3a57434d605412c82946ec0f6f80b2a I0628 23:54:58.380786 30213 raft_consensus.cc:2019] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 0 FOLLOWER]: Advancing to term 1 I0628 23:54:58.384444 30213 raft_consensus.cc:1677] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bea06ed38f24dd3b85c977539f7d5f5 in term 1. I0628 23:54:58.385431 30030 leader_election.cc:347] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [CANDIDATE]: Term 1 election: Vote granted by peer c3a57434d605412c82946ec0f6f80b2a I0628 23:54:58.385617 30030 leader_election.cc:248] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:54:58.386307 30339 raft_consensus.cc:1859] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:54:58.387807 30339 raft_consensus.cc:464] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 LEADER]: Becoming Leader. State: Replica: 8bea06ed38f24dd3b85c977539f7d5f5, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:54:58.388263 30339 consensus_queue.cc:145] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } I0628 23:54:58.424118 30286 catalog_manager.cc:1673] Tablet: 81bb583ab6264dd3bed571d157bdc78e reported consensus state change. New consensus state: current_term: 1 leader_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" config { opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } } W0628 23:54:58.464993 30348 raft_consensus_state.cc:541] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:54:58.530340 30213 raft_consensus.cc:869] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 1 FOLLOWER]: Refusing update from remote peer 8bea06ed38f24dd3b85c977539f7d5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:54:58.532102 30339 consensus_queue.cc:578] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [LEADER]: Connected to new peer: Peer: c3a57434d605412c82946ec0f6f80b2a, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:54:58.568624 30348 raft_consensus_state.cc:534] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 3 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==30025==WARNING: Can't read from symbolizer at fd 28 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==30025==WARNING: Can't read from symbolizer at fd 28 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==30142==WARNING: external symbolizer didn't start up correctly! ==30142==WARNING: Can't write to symbolizer at fd 30 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==30025==WARNING: external symbolizer didn't start up correctly! ==30025==WARNING: Can't write to symbolizer at fd 31 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==30142==WARNING: external symbolizer didn't start up correctly! ==30142==WARNING: Can't write to symbolizer at fd 30 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==30142==WARNING: external symbolizer didn't start up correctly! ==30142==WARNING: Can't write to symbolizer at fd 30 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==30025==WARNING: Can't read from symbolizer at fd 28 ==30025==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7d700006f828 by thread T127: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f829 by thread T126: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==30142==WARNING: Can't read from symbolizer at fd 27 ================== WARNING: ThreadSanitizer: data race (pid=30025) Atomic write of size 8 at 0x7d700006f820 by thread T127: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f821 by thread T126: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Atomic write of size 4 at 0x7d700006f8f0 by thread T127: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T126: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Atomic write of size 8 at 0x7d700006f840 by thread T127: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f841 by thread T126: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7d700006f8b8 by thread T127: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8b8 by thread T125: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7d700006f830 by thread T128: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f831 by thread T127: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7d700006f850 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f850 by thread T126: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) ==30142==WARNING: Failed to use and restart external symbolizer! Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7d700006f848 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f848 by thread T122: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7d700006f838 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f839 by thread T128: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadd1) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7d700006f840 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f841 by thread T122: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) ================== #3 (libkudu_util.so+0x000000224304) WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7d700006f808 by thread T127: #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #0 (libtablet.so+0x0000002994b2) #6 (libkudu_util.so+0x000000224632) #1 (libtablet.so+0x000000299394) #7 (libtablet.so+0x00000023ae76) #2 (libtablet.so+0x000000298916) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #3 (libtablet.so+0x000000297af3) #10 (libtablet.so+0x000000234508) #4 (libtablet.so+0x000000297dca) #11 (libtablet.so+0x00000023444a) #5 (libtablet.so+0x000000296aeb) #12 (libmaster_rpc.so+0x000000011c31) #6 (libtablet.so+0x0000002965d3) #13 (libconsensus.so+0x000000096280) #7 (libtablet.so+0x00000028cafd) #14 (libconsensus.so+0x0000000f0f2d) #8 (libtablet.so+0x00000028936b) #15 (libconsensus.so+0x0000000d93bb) #9 (libtablet.so+0x0000001dcebd) #16 (libconsensus.so+0x0000000d8c4f) #10 (libtablet.so+0x0000001dde5d) #17 (libtserver.so+0x00000012dafc) #11 (libtablet.so+0x0000001ddcaf) #18 (libconsensus_proto.so+0x00000006d374) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #19 (libconsensus_proto.so+0x00000006d18e) #14 (libtablet.so+0x00000023d2c6) #20 (libkrpc.so+0x000000121d48) #15 (libtablet.so+0x00000023d255) #21 (libkrpc.so+0x000000121948) #16 (libtablet.so+0x00000023d1ea) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #17 (libconsensus.so+0x0000000befdd) #24 (libkrpc.so+0x0000001253da) #18 (libconsensus.so+0x0000000c5726) #25 (libkrpc.so+0x000000125383) #19 (libconsensus.so+0x0000000c568e) #26 (libkrpc.so+0x0000001251a9) #20 (libconsensus.so+0x0000000c5643) #27 (libkrpc.so+0x0000000cce71) #21 (libconsensus.so+0x0000000c5401) #28 (libkudu_util.so+0x00000021b55f) #22 (libkrpc.so+0x0000000cce71) #29 (kudu-tserver+0x000000421829) #23 (libkudu_util.so+0x000000226bcd) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 #24 (libkudu_util.so+0x000000224fcf) at: #0 (kudu-tserver+0x00000044a123) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #1 (libkudu_util.so+0x00000021ade7) #27 (libkudu_util.so+0x000000229303) #2 (libkudu_util.so+0x0000002262ae) #28 (libkudu_util.so+0x000000229101) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #29 (libkrpc.so+0x0000000cce71) #5 (libkudu_util.so+0x000000224704) #30 (libkudu_util.so+0x00000021b55f) #6 (libkudu_util.so+0x000000224632) #31 (kudu-tserver+0x000000421829) #7 (libtablet.so+0x00000023ae76) Previous read of size 8 at 0x7d700006f808 by thread T128: #0 (libtablet.so+0x0000001fb1cf) #8 (libtablet.so+0x00000023908d) #1 (libtablet.so+0x0000001fad02) #9 (libtablet.so+0x000000234615) #2 (libtablet.so+0x000000295cf2) #10 (libtablet.so+0x000000234508) #3 (libtablet.so+0x00000028c6b7) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #4 (libtablet.so+0x000000289219) #13 (libconsensus.so+0x000000096280) #5 (libtablet.so+0x0000001dcebd) #14 (libconsensus.so+0x0000000f0f2d) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #15 (libconsensus.so+0x0000000d93bb) #8 (libtablet.so+0x0000002438f1) #16 (libconsensus.so+0x0000000d8c4f) #9 (libtablet.so+0x00000023b2f5) #17 (libtserver.so+0x00000012dafc) #10 (libtablet.so+0x00000023d2c6) #18 (libconsensus_proto.so+0x00000006d374) #11 (libtablet.so+0x00000023d255) #19 (libconsensus_proto.so+0x00000006d18e) #12 (libtablet.so+0x00000023d1ea) #20 (libkrpc.so+0x000000121d48) #13 (libconsensus.so+0x0000000befdd) #21 (libkrpc.so+0x000000121948) #14 (libconsensus.so+0x0000000c5726) #22 (libkrpc.so+0x00000012369d) #15 (libconsensus.so+0x0000000c568e) #23 (libkrpc.so+0x000000125476) #16 (libconsensus.so+0x0000000c5643) #24 (libkrpc.so+0x0000001253da) #17 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x000000125383) #18 (libkrpc.so+0x0000000cce71) #26 (libkrpc.so+0x0000001251a9) #19 (libkudu_util.so+0x000000226bcd) #27 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000021b55f) #21 (libkudu_util.so+0x00000022941e) #29 (kudu-tserver+0x000000421829) #22 (libkudu_util.so+0x00000022937b) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) ================== #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) ================== WARNING: ThreadSanitizer: data race (pid=30142) #9 (libtablet.so+0x000000234615) Atomic write of size 8 at 0x7d700006f820 by thread T123: #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #0 (kudu-tserver+0x000000466ec0) #12 (libmaster_rpc.so+0x000000011c31) #1 (libtablet.so+0x000000227502) #13 (libconsensus.so+0x000000096280) #2 (libtablet.so+0x0000002972a5) #14 (libconsensus.so+0x0000000f0f2d) #3 (libtablet.so+0x000000297240) #15 (libconsensus.so+0x0000000f0017) #4 (libtablet.so+0x00000028bc4a) #16 (libconsensus.so+0x0000000d76b1) #5 (libtablet.so+0x000000298a34) #17 (libconsensus.so+0x0000000d73d0) #6 (libtablet.so+0x0000002979d1) #18 (libconsensus.so+0x0000000d79cd) #7 (libtablet.so+0x000000296aeb) #19 (libconsensus.so+0x0000000a97f2) #8 (libtablet.so+0x0000002965d3) #20 (libconsensus.so+0x0000000ba5e8) #9 (libtablet.so+0x00000028cafd) #21 (libconsensus.so+0x0000000ba558) #10 (libtablet.so+0x00000028936b) #22 (libconsensus.so+0x0000000ba4d3) #11 (libtablet.so+0x0000001dcebd) #23 (libconsensus.so+0x0000000befdd) #12 (libtablet.so+0x0000001dde5d) #24 (libconsensus.so+0x0000000c5726) #13 (libtablet.so+0x0000001ddcaf) #25 (libconsensus.so+0x0000000c568e) #14 (libtablet.so+0x0000002438f1) #26 (libconsensus.so+0x0000000c5643) #15 (libtablet.so+0x00000023b2f5) #27 (libconsensus.so+0x0000000c5401) #16 (libtablet.so+0x00000023d2c6) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #17 (libtablet.so+0x00000023d255) #30 (libkudu_util.so+0x000000224fcf) #18 (libtablet.so+0x00000023d1ea) #31 (libkudu_util.so+0x00000022941e) #19 (libconsensus.so+0x0000000befdd) #32 (libkudu_util.so+0x00000022937b) #20 (libconsensus.so+0x0000000c5726) #33 (libkudu_util.so+0x000000229303) #21 (libconsensus.so+0x0000000c568e) #34 (libkudu_util.so+0x000000229101) #22 (libconsensus.so+0x0000000c5643) #35 (libkrpc.so+0x0000000cce71) #23 (libconsensus.so+0x0000000c5401) #36 (libkudu_util.so+0x00000021b55f) #24 (libkrpc.so+0x0000000cce71) #37 (kudu-tserver+0x000000421829) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #27 (libkudu_util.so+0x00000022941e) #0 (kudu-tserver+0x00000044a123) #28 (libkudu_util.so+0x00000022937b) #1 (libkudu_util.so+0x00000021ade7) #29 (libkudu_util.so+0x000000229303) #2 (libkudu_util.so+0x0000002262ae) #30 (libkudu_util.so+0x000000229101) #3 (libkudu_util.so+0x000000224304) #31 (libkrpc.so+0x0000000cce71) #4 (libkudu_util.so+0x000000224902) #32 (libkudu_util.so+0x00000021b55f) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #33 (kudu-tserver+0x000000421829) #7 (libtablet.so+0x00000023ae76) Previous read of size 1 at 0x7d700006f821 by thread T122: #0 (libtablet.so+0x0000001fc02e) #8 (libtablet.so+0x00000023908d) #1 (libtablet.so+0x0000001fbf59) #9 (libtablet.so+0x000000234615) #2 (libtablet.so+0x0000001fbbd2) #10 (libtablet.so+0x000000234508) #3 (libtablet.so+0x0000001fb7ea) #11 (libtablet.so+0x00000023444a) #4 (libtablet.so+0x0000001fb609) #12 (libmaster_rpc.so+0x000000011c31) #5 (libtablet.so+0x0000001fb3e7) #13 (libconsensus.so+0x000000096280) #6 (libtablet.so+0x0000001fad7b) #14 (libconsensus.so+0x0000000f0f2d) #7 (libtablet.so+0x000000295cf2) #15 (libconsensus.so+0x0000000f0017) #8 (libtablet.so+0x00000028c6b7) #16 (libconsensus.so+0x0000000d76b1) #9 (libtablet.so+0x000000289219) #17 (libconsensus.so+0x0000000d73d0) #10 (libtablet.so+0x0000001dcebd) #18 (libconsensus.so+0x0000000d79cd) #11 (libtablet.so+0x0000001dde5d) #19 (libconsensus.so+0x0000000a97f2) #12 (libtablet.so+0x0000001ddcaf) #20 (libconsensus.so+0x0000000ba5e8) #13 (libtablet.so+0x0000002438f1) #21 (libconsensus.so+0x0000000ba558) #14 (libtablet.so+0x00000023b2f5) #22 (libconsensus.so+0x0000000ba4d3) #15 (libtablet.so+0x00000023d2c6) #23 (libconsensus.so+0x0000000befdd) #16 (libtablet.so+0x00000023d255) #24 (libconsensus.so+0x0000000c5726) #17 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000befdd) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #19 (libconsensus.so+0x0000000c5726) #29 (libkudu_util.so+0x000000226bcd) #20 (libconsensus.so+0x0000000c568e) #30 (libkudu_util.so+0x000000224fcf) #21 (libconsensus.so+0x0000000c5643) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #22 (libconsensus.so+0x0000000c5401) #33 (libkudu_util.so+0x000000229303) #23 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x000000229101) #24 (libkudu_util.so+0x000000226bcd) #35 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000224fcf) #36 (libkudu_util.so+0x00000021b55f) #26 (libkudu_util.so+0x00000022941e) #37 (kudu-tserver+0x000000421829) Thread T125 #27 (libkudu_util.so+0x00000022937b) 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #28 (libkudu_util.so+0x000000229303) #1 (libkudu_util.so+0x00000021ade7) #29 (libkudu_util.so+0x000000229101) #2 (libkudu_util.so+0x0000002262ae) #30 (libkrpc.so+0x0000000cce71) #3 (libkudu_util.so+0x000000224304) #31 (libkudu_util.so+0x00000021b55f) #4 (libkudu_util.so+0x000000224902) #32 (kudu-tserver+0x000000421829) #5 (libkudu_util.so+0x000000224704) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #6 (libkudu_util.so+0x000000224632) #1 (libkudu_util.so+0x0000001d048f) #7 (libtablet.so+0x00000023ae76) #2 (libkudu_util.so+0x0000001d039a) #8 (libtablet.so+0x00000023908d) #3 (libkudu_util.so+0x0000001d11a9) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #4 (libkudu_util.so+0x0000001d0e5f) #11 (libtablet.so+0x00000023444a) #5 (libkudu_util.so+0x0000001cc3ab) #12 (libmaster_rpc.so+0x000000011c31) #6 (libkudu_util.so+0x0000001cb1d4) #13 (libconsensus.so+0x000000096280) #7 (libkudu_util.so+0x0000001cba25) #14 (libconsensus.so+0x0000000f0f2d) #8 (libtablet.so+0x000000295539) #15 (libconsensus.so+0x0000000f00a1) #9 (libtablet.so+0x0000002988dc) #16 (libconsensus.so+0x0000000d76b1) #10 (libtablet.so+0x000000297af3) #17 (libconsensus.so+0x0000000d73d0) #11 (libtablet.so+0x000000296aeb) #18 (libconsensus.so+0x0000000d79cd) #12 (libtablet.so+0x0000002965d3) #19 (libconsensus.so+0x0000000a97f2) #13 (libtablet.so+0x00000028cafd) #20 (libconsensus.so+0x0000000ba5e8) #14 (libtablet.so+0x00000028936b) #21 (libconsensus.so+0x0000000ba558) #15 (libtablet.so+0x0000001dcebd) #22 (libconsensus.so+0x0000000ba4d3) #16 (libtablet.so+0x0000001dde5d) #23 (libconsensus.so+0x0000000befdd) #17 (libtablet.so+0x0000001ddcaf) #24 (libconsensus.so+0x0000000c5726) #18 (libtablet.so+0x0000002438f1) #25 (libconsensus.so+0x0000000c568e) #19 (libtablet.so+0x00000023b2f5) #26 (libconsensus.so+0x0000000c5643) #20 (libtablet.so+0x00000023d2c6) #27 (libconsensus.so+0x0000000c5401) #21 (libtablet.so+0x00000023d255) #28 (libkrpc.so+0x0000000cce71) #22 (libtablet.so+0x00000023d1ea) #29 (libkudu_util.so+0x000000226bcd) #23 (libconsensus.so+0x0000000befdd) #30 (libkudu_util.so+0x000000224fcf) #24 (libconsensus.so+0x0000000c5726) #31 (libkudu_util.so+0x00000022941e) #25 (libconsensus.so+0x0000000c568e) #32 (libkudu_util.so+0x00000022937b) #26 (libconsensus.so+0x0000000c5643) #33 (libkudu_util.so+0x000000229303) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x000000229101) #29 (libkudu_util.so+0x000000226bcd) #35 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000224fcf) #36 (libkudu_util.so+0x00000021b55f) #31 (libkudu_util.so+0x00000022941e) #37 (kudu-tserver+0x000000421829) #32 (libkudu_util.so+0x00000022937b) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) #33 (libkudu_util.so+0x000000229303) ================== #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7d500002fd38 by thread T127: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500002fd38 by thread T128: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) ================== WARNING: ThreadSanitizer: data race (pid=30142) #15 (libtablet.so+0x00000023d255) Write of size 8 at 0x7d700006f8b8 by thread T123: #16 (libtablet.so+0x00000023d1ea) #0 (libtablet.so+0x000000298a88) #17 (libconsensus.so+0x0000000befdd) #1 (libtablet.so+0x0000002979d1) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #2 (libtablet.so+0x000000296aeb) #20 (libconsensus.so+0x0000000c5643) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #21 (libconsensus.so+0x0000000c5401) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #22 (libkrpc.so+0x0000000cce71) #7 (libtablet.so+0x0000001dde5d) #23 (libkudu_util.so+0x000000226bcd) #8 (libtablet.so+0x0000001ddcaf) #24 (libkudu_util.so+0x000000224fcf) #9 (libtablet.so+0x0000002438f1) #25 (libkudu_util.so+0x00000022941e) #10 (libtablet.so+0x00000023b2f5) #26 (libkudu_util.so+0x00000022937b) #11 (libtablet.so+0x00000023d2c6) #27 (libkudu_util.so+0x000000229303) #12 (libtablet.so+0x00000023d255) #28 (libkudu_util.so+0x000000229101) #13 (libtablet.so+0x00000023d1ea) #29 (libkrpc.so+0x0000000cce71) #14 (libconsensus.so+0x0000000befdd) #30 (libkudu_util.so+0x00000021b55f) #15 (libconsensus.so+0x0000000c5726) #31 (kudu-tserver+0x000000421829) #16 (libconsensus.so+0x0000000c568e) Location is heap block of size 456 at 0x7d500002fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #17 (libconsensus.so+0x0000000c5643) #1 (libtablet.so+0x0000001db43e) #18 (libconsensus.so+0x0000000c5401) #2 (libtablet.so+0x000000210662) #19 (libkrpc.so+0x0000000cce71) #3 (libtablet.so+0x00000020f9b7) #20 (libkudu_util.so+0x000000226bcd) #4 (libtablet.so+0x00000020f54e) #21 (libkudu_util.so+0x000000224fcf) #5 (libtserver.so+0x00000014bc0e) #22 (libkudu_util.so+0x00000022941e) #6 (libtserver.so+0x000000157d97) #23 (libkudu_util.so+0x00000022937b) #7 (libtserver.so+0x000000157ce3) #24 (libkudu_util.so+0x000000229303) #8 (libtserver.so+0x000000157c63) #25 (libkudu_util.so+0x000000229101) #9 (libtserver.so+0x000000157a21) #26 (libkrpc.so+0x0000000cce71) #10 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #11 (libkudu_util.so+0x000000226bcd) #28 (kudu-tserver+0x000000421829) #12 (libkudu_util.so+0x000000224fcf) Previous read of size 8 at 0x7d700006f8b8 by thread T122: #0 (libtablet.so+0x0000001fb3fb) #13 (libkudu_util.so+0x00000022941e) #1 (libtablet.so+0x0000001fad7b) #14 (libkudu_util.so+0x00000022937b) #2 (libtablet.so+0x000000295cf2) #15 (libkudu_util.so+0x000000229303) #3 (libtablet.so+0x00000028c6b7) #16 (libkudu_util.so+0x000000229101) #4 (libtablet.so+0x000000289219) #17 (libkrpc.so+0x0000000cce71) #5 (libtablet.so+0x0000001dcebd) #18 (libkudu_util.so+0x00000021b55f) #6 (libtablet.so+0x0000001dde5d) #19 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' #7 (libtablet.so+0x0000001ddcaf) (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #8 (libtablet.so+0x0000002438f1) #1 (libkudu_util.so+0x00000021ade7) #9 (libtablet.so+0x00000023b2f5) #2 (libkudu_util.so+0x0000002262ae) #10 (libtablet.so+0x00000023d2c6) #3 (libkudu_util.so+0x000000224304) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #4 (libkudu_util.so+0x000000224902) #13 (libconsensus.so+0x0000000befdd) #5 (libkudu_util.so+0x000000224704) #14 (libconsensus.so+0x0000000c5726) #6 (libkudu_util.so+0x000000224632) #15 (libconsensus.so+0x0000000c568e) #7 (libtablet.so+0x00000023ae76) #16 (libconsensus.so+0x0000000c5643) #8 (libtablet.so+0x00000023908d) #17 (libconsensus.so+0x0000000c5401) #9 (libtablet.so+0x000000234615) #18 (libkrpc.so+0x0000000cce71) #10 (libtablet.so+0x000000234508) #19 (libkudu_util.so+0x000000226bcd) #11 (libtablet.so+0x00000023444a) #20 (libkudu_util.so+0x000000224fcf) #12 (libmaster_rpc.so+0x000000011c31) #21 (libkudu_util.so+0x00000022941e) #13 (libconsensus.so+0x000000096280) #22 (libkudu_util.so+0x00000022937b) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #23 (libkudu_util.so+0x000000229303) #16 (libconsensus.so+0x0000000d76b1) #24 (libkudu_util.so+0x000000229101) #17 (libconsensus.so+0x0000000d73d0) #25 (libkrpc.so+0x0000000cce71) #18 (libconsensus.so+0x0000000d79cd) #26 (libkudu_util.so+0x00000021b55f) #19 (libconsensus.so+0x0000000a97f2) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #20 (libconsensus.so+0x0000000ba5e8) #0 (kudu-tserver+0x000000440858) #21 (libconsensus.so+0x0000000ba558) #1 (libkudu_util.so+0x0000001d048f) #22 (libconsensus.so+0x0000000ba4d3) #2 (libkudu_util.so+0x0000001d039a) #23 (libconsensus.so+0x0000000befdd) #3 (libkudu_util.so+0x0000001d11a9) #24 (libconsensus.so+0x0000000c5726) #4 (libkudu_util.so+0x0000001d0e5f) #25 (libconsensus.so+0x0000000c568e) #5 (libkudu_util.so+0x0000001cc3ab) #26 (libconsensus.so+0x0000000c5643) #6 (libkudu_util.so+0x0000001cb1d4) #27 (libconsensus.so+0x0000000c5401) #7 (libkudu_util.so+0x0000001cba25) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #8 (libtablet.so+0x000000295539) #30 (libkudu_util.so+0x000000224fcf) #9 (libtablet.so+0x0000002988dc) #31 (libkudu_util.so+0x00000022941e) #10 (libtablet.so+0x000000297af3) #32 (libkudu_util.so+0x00000022937b) #11 (libtablet.so+0x000000296aeb) #33 (libkudu_util.so+0x000000229303) #12 (libtablet.so+0x0000002965d3) #34 (libkudu_util.so+0x000000229101) #13 (libtablet.so+0x00000028cafd) #35 (libkrpc.so+0x0000000cce71) #14 (libtablet.so+0x00000028936b) #36 (libkudu_util.so+0x00000021b55f) #15 (libtablet.so+0x0000001dcebd) #37 (kudu-tserver+0x000000421829) #16 (libtablet.so+0x0000001dde5d) Thread T128 'apply [worker]-' #17 (libtablet.so+0x0000001ddcaf) (tid=30355, running) created by thread T123 at: #18 (libtablet.so+0x0000002438f1) #0 (kudu-tserver+0x00000044a123) #19 (libtablet.so+0x00000023b2f5) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #20 (libtablet.so+0x00000023d2c6) #3 (libkudu_util.so+0x000000224304) #21 (libtablet.so+0x00000023d255) #4 (libkudu_util.so+0x000000224902) #22 (libtablet.so+0x00000023d1ea) #5 (libkudu_util.so+0x000000224704) #23 (libconsensus.so+0x0000000befdd) #6 (libkudu_util.so+0x000000224632) #24 (libconsensus.so+0x0000000c5726) #7 (libtablet.so+0x00000023ae76) #25 (libconsensus.so+0x0000000c568e) #8 (libtablet.so+0x00000023908d) #26 (libconsensus.so+0x0000000c5643) #9 (libtablet.so+0x000000234615) #27 (libconsensus.so+0x0000000c5401) #10 (libtablet.so+0x000000234508) #28 (libkrpc.so+0x0000000cce71) #11 (libtablet.so+0x00000023444a) #29 (libkudu_util.so+0x000000226bcd) #12 (libmaster_rpc.so+0x000000011c31) #30 (libkudu_util.so+0x000000224fcf) #13 (libconsensus.so+0x000000096280) #31 (libkudu_util.so+0x00000022941e) #14 (libconsensus.so+0x0000000f0f2d) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #15 (libconsensus.so+0x0000000f0017) #34 (libkudu_util.so+0x000000229101) #16 (libconsensus.so+0x0000000d76b1) #35 (libkrpc.so+0x0000000cce71) #17 (libconsensus.so+0x0000000d73d0) #36 (libkudu_util.so+0x00000021b55f) #18 (libconsensus.so+0x0000000d79cd) #37 (kudu-tserver+0x000000421829) #19 (libconsensus.so+0x0000000a97f2) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #20 (libconsensus.so+0x0000000ba5e8) #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #21 (libconsensus.so+0x0000000ba558) #2 (libkudu_util.so+0x0000002262ae) #22 (libconsensus.so+0x0000000ba4d3) #3 (libkudu_util.so+0x000000224304) #23 (libconsensus.so+0x0000000befdd) #4 (libkudu_util.so+0x000000224902) #24 (libconsensus.so+0x0000000c5726) #5 (libkudu_util.so+0x000000224704) #25 (libconsensus.so+0x0000000c568e) #6 (libkudu_util.so+0x000000224632) #26 (libconsensus.so+0x0000000c5643) #7 (libtablet.so+0x00000023ae76) #27 (libconsensus.so+0x0000000c5401) #8 (libtablet.so+0x00000023908d) #28 (libkrpc.so+0x0000000cce71) #9 (libtablet.so+0x000000234615) #29 (libkudu_util.so+0x000000226bcd) #10 (libtablet.so+0x000000234508) #30 (libkudu_util.so+0x000000224fcf) #11 (libtablet.so+0x00000023444a) #31 (libkudu_util.so+0x00000022941e) #12 (libmaster_rpc.so+0x000000011c31) #32 (libkudu_util.so+0x00000022937b) #13 (libconsensus.so+0x000000096280) #33 (libkudu_util.so+0x000000229303) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #34 (libkudu_util.so+0x000000229101) #16 (libconsensus.so+0x0000000d8c4f) #35 (libkrpc.so+0x0000000cce71) #17 (libtserver.so+0x00000012dafc) #36 (libkudu_util.so+0x00000021b55f) #18 (libconsensus_proto.so+0x00000006d374) #37 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' #19 (libconsensus_proto.so+0x00000006d18e) (tid=30333, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #20 (libkrpc.so+0x000000121d48) #1 (libkudu_util.so+0x00000021ade7) #21 (libkrpc.so+0x000000121948) #2 (libkudu_util.so+0x0000002262ae) #22 (libkrpc.so+0x00000012369d) #3 (libkudu_util.so+0x000000224304) #23 (libkrpc.so+0x000000125476) #4 (libkudu_util.so+0x000000224902) #24 (libkrpc.so+0x0000001253da) #5 (libkudu_util.so+0x000000224704) #25 (libkrpc.so+0x000000125383) #6 (libtserver.so+0x00000014cd50) #26 (libkrpc.so+0x0000001251a9) #7 (libtserver.so+0x00000012b940) #27 (libkrpc.so+0x0000000cce71) #8 (libtserver_admin_proto.so+0x000000028a24) #28 (libkudu_util.so+0x00000021b55f) #9 (libtserver_admin_proto.so+0x00000002883e) #29 (kudu-tserver+0x000000421829) #10 (libkrpc.so+0x000000121d48) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #11 (libkrpc.so+0x000000121948) #0 (kudu-tserver+0x00000044a123) #12 (libkrpc.so+0x00000012369d) #1 (libkudu_util.so+0x00000021ade7) #13 (libkrpc.so+0x000000125476) #2 (libkudu_util.so+0x0000002262ae) #14 (libkrpc.so+0x0000001253da) #3 (libkudu_util.so+0x000000224304) #15 (libkrpc.so+0x000000125383) #4 (libkudu_util.so+0x000000224902) #16 (libkrpc.so+0x0000001251a9) #5 (libkudu_util.so+0x000000224704) #17 (libkrpc.so+0x0000000cce71) #6 (libkudu_util.so+0x000000224632) #18 (libkudu_util.so+0x00000021b55f) #7 (libtablet.so+0x00000023ae76) #19 (kudu-tserver+0x000000421829) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) ================== #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Atomic write of size 4 at 0x7d700006f8f0 by thread T123: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T122: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7d700006f8a8 by thread T124: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8a8 by thread T123: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000002960ab) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000298858) #5 (libtablet.so+0x0000002979a4) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7d700006f808 by thread T122: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f808 by thread T124: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7d500002fd38 by thread T122: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500002fd38 by thread T124: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500002fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=30332, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Atomic write of size 4 at 0x7dc00000ec70 by thread T123: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dc00000ec70 by thread T122: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 1 at 0x7dc00000ebc1 by thread T124: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000297dca) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dc00000ebc0 by thread T122: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 4 at 0x7d900004fbe0 by thread T124: #0 (libtablet.so+0x000000299980) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900004fbe0 by thread T123: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299980) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7d900004fbc0 by thread T124: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) ================== WARNING: ThreadSanitizer: data race (pid=30025) Atomic write of size 4 at 0x7dd000001de8 by thread T128: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000297dca) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000001de8 by thread T126: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900004fbc0 by thread T123: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 1 at 0x7dc00000ec89 by thread T124: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000297dca) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dc00000ec88 by thread T122: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb826) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7dc00000ed00 by thread T124: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dc00000ed00 by thread T125: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Atomic write of size 4 at 0x7dc00000ed68 by thread T124: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7f82607c1ea8 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7f82607c1ea8 by thread T128: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f82607c1000 allocated by thread T128: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libtablet.so+0x000000297dca) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #5 (libkudu_util.so+0x0000001cc3ab) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #6 (libkudu_util.so+0x0000001cb1d4) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #7 (libkudu_util.so+0x0000001cba25) #13 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x000000295539) #14 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000029748b) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #10 (libtablet.so+0x00000028bd5a) #17 (libtablet.so+0x00000023d1ea) #11 (libtablet.so+0x000000296ec3) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #12 (libtablet.so+0x0000002968fc) #20 (libconsensus.so+0x0000000c568e) #13 (libtablet.so+0x00000029659b) #21 (libconsensus.so+0x0000000c5643) #14 (libtablet.so+0x00000028cafd) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #15 (libtablet.so+0x00000028936b) #24 (libkudu_util.so+0x000000226bcd) #16 (libtablet.so+0x0000001dcebd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dc00000ed68 by thread T125: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #23 (libtablet.so+0x00000023d1ea) #20 (libtablet.so+0x00000023b2f5) #24 (libconsensus.so+0x0000000befdd) #21 (libtablet.so+0x00000023d2c6) #25 (libconsensus.so+0x0000000c5726) #22 (libtablet.so+0x00000023d255) #26 (libconsensus.so+0x0000000c568e) #23 (libtablet.so+0x00000023d1ea) #27 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #28 (libconsensus.so+0x0000000c5401) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #29 (libkrpc.so+0x0000000cce71) #28 (libconsensus.so+0x0000000c5401) #30 (libkudu_util.so+0x000000226bcd) #29 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Atomic write of size 8 at 0x7f82607c1ea0 by thread T125: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f82607c1ea1 by thread T128: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f82607c1000 allocated by thread T128: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7f82607c1f20 by thread T125: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f82607c1f20 by thread T128: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f82607c1000 allocated by thread T128: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7dc00000ebc8 by thread T122: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x000000297d2b) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dc00000ebc9 by thread T123: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) ================== #6 (libkudu_util.so+0x0000001cb1d4) WARNING: ThreadSanitizer: data race (pid=30025) Atomic write of size 4 at 0x7f82607c1f58 by thread T125: #0 (kudu-tserver+0x000000466be0) #7 (libkudu_util.so+0x0000001cba25) #1 (libtserver.so+0x0000001573d5) #8 (libtablet.so+0x000000295539) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #9 (libtablet.so+0x00000029748b) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x00000028bd5a) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x000000296ec3) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000002968fc) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000029659b) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #15 (libtablet.so+0x00000028936b) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #16 (libtablet.so+0x0000001dcebd) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #17 (libtablet.so+0x0000001dde5d) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #18 (libtablet.so+0x0000001ddcaf) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #19 (libtablet.so+0x0000002438f1) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #20 (libtablet.so+0x00000023b2f5) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #21 (libtablet.so+0x00000023d2c6) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f82607c1f58 by thread T128: #22 (libtablet.so+0x00000023d255) #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #23 (libtablet.so+0x00000023d1ea) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #24 (libconsensus.so+0x0000000befdd) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #25 (libconsensus.so+0x0000000c5726) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #26 (libconsensus.so+0x0000000c568e) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #27 (libconsensus.so+0x0000000c5643) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #28 (libconsensus.so+0x0000000c5401) #14 (libconsensus.so+0x0000000c568e) #29 (libkrpc.so+0x0000000cce71) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #33 (libkudu_util.so+0x00000022937b) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #34 (libkudu_util.so+0x000000229303) #26 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f82607c1000 allocated by thread T128: #0 (kudu-tserver+0x000000440858) #35 (libkudu_util.so+0x000000229101) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #36 (libkrpc.so+0x0000000cce71) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #37 (libkudu_util.so+0x00000021b55f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #38 (kudu-tserver+0x000000421829) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #1 (libkudu_util.so+0x00000021ade7) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #2 (libkudu_util.so+0x0000002262ae) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #3 (libkudu_util.so+0x000000224304) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #4 (libkudu_util.so+0x000000224902) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #5 (libkudu_util.so+0x000000224704) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #6 (libkudu_util.so+0x000000224632) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #7 (libtablet.so+0x00000023ae76) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #8 (libtablet.so+0x00000023908d) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #9 (libtablet.so+0x000000234615) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #10 (libtablet.so+0x000000234508) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #11 (libtablet.so+0x00000023444a) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) #12 (libmaster_rpc.so+0x000000011c31) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #13 (libconsensus.so+0x000000096280) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #14 (libconsensus.so+0x0000000f0f2d) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #15 (libconsensus.so+0x0000000d93bb) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #16 (libconsensus.so+0x0000000d8c4f) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #17 (libtserver.so+0x00000012dafc) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #18 (libconsensus_proto.so+0x00000006d374) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #19 (libconsensus_proto.so+0x00000006d18e) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #20 (libkrpc.so+0x000000121d48) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #21 (libkrpc.so+0x000000121948) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) #22 (libkrpc.so+0x00000012369d) ================== #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7dd000001c30 by thread T127: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000001c31 by thread T126: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) ================== WARNING: ThreadSanitizer: data race (pid=30142) Atomic write of size 8 at 0x7f4ccce55038 by thread T125: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #4 (libkudu_util.so+0x000000224902) #2 (libtablet.so+0x0000002972a5) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f4ccce55039 by thread T122: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f4ccce55000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000298e36) #11 (libtablet.so+0x000000297b40) #12 (libtablet.so+0x000000296aeb) #13 (libtablet.so+0x0000002965d3) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 1 at 0x7dc000000e91 by thread T128: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dc000000e90 by thread T125: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Atomic write of size 8 at 0x7dd000001d18 by thread T126: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000297dca) #9 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x0000002965d3) #11 (libtablet.so+0x00000028cafd) #12 (libtablet.so+0x00000028936b) #13 (libtablet.so+0x0000001dcebd) #14 (libtablet.so+0x0000001dde5d) #15 (libtablet.so+0x0000001ddcaf) #16 (libtablet.so+0x0000002438f1) #17 (libtablet.so+0x00000023b2f5) #18 (libtablet.so+0x00000023d2c6) #19 (libtablet.so+0x00000023d255) #20 (libtablet.so+0x00000023d1ea) #21 (libconsensus.so+0x0000000befdd) #22 (libconsensus.so+0x0000000c5726) #23 (libconsensus.so+0x0000000c568e) #24 (libconsensus.so+0x0000000c5643) #25 (libconsensus.so+0x0000000c5401) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x000000226bcd) #28 (libkudu_util.so+0x000000224fcf) #29 (libkudu_util.so+0x00000022941e) #30 (libkudu_util.so+0x00000022937b) #31 (libkudu_util.so+0x000000229303) #32 (libkudu_util.so+0x000000229101) #33 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x00000021b55f) #35 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000001d19 by thread T125: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7dd000001d90 by thread T126: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd000001d90 by thread T125: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7f4ccc8aa3d8 by thread T122: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f4ccc8aa3d9 by thread T124: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f4ccc883000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7f4ccc8aa3d0 by thread T122: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7f4ccc8aa3d1 by thread T124: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f4ccc883000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7f4ccc8aa3c8 by thread T122: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7f4ccc8aa3c8 by thread T124: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f4ccc883000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Atomic write of size 8 at 0x7f4ccc8aa3c0 by thread T122: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f4ccc8aa3c1 by thread T124: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f4ccc883000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7f4ccc8aa448 by thread T122: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f4ccc8aa448 by thread T124: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f4ccc883000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7f4ccc8aa438 by thread T122: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f4ccc8aa438 by thread T124: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f4ccc883000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7dd000001c48 by thread T127: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dd000001c49 by thread T125: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7dd000001d40 by thread T128: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dd000001d40 by thread T127: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7dc00000ecc0 by thread T124: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dc00000ecc0 by thread T122: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7dc00000ecb8 by thread T124: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dc00000ecb9 by thread T122: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Atomic write of size 8 at 0x7dc00000ec98 by thread T124: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000297dca) #9 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x0000002965d3) #11 (libtablet.so+0x00000028cafd) #12 (libtablet.so+0x00000028936b) #13 (libtablet.so+0x0000001dcebd) #14 (libtablet.so+0x0000001dde5d) #15 (libtablet.so+0x0000001ddcaf) #16 (libtablet.so+0x0000002438f1) #17 (libtablet.so+0x00000023b2f5) #18 (libtablet.so+0x00000023d2c6) #19 (libtablet.so+0x00000023d255) #20 (libtablet.so+0x00000023d1ea) #21 (libconsensus.so+0x0000000befdd) #22 (libconsensus.so+0x0000000c5726) #23 (libconsensus.so+0x0000000c568e) #24 (libconsensus.so+0x0000000c5643) #25 (libconsensus.so+0x0000000c5401) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x000000226bcd) #28 (libkudu_util.so+0x000000224fcf) #29 (libkudu_util.so+0x00000022941e) #30 (libkudu_util.so+0x00000022937b) #31 (libkudu_util.so+0x000000229303) #32 (libkudu_util.so+0x000000229101) #33 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x00000021b55f) #35 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dc00000ec99 by thread T125: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30369, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7dc00000ed30 by thread T124: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dc00000ed30 by thread T122: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T124: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7dd000001d20 by thread T126: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x000000297d2b) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000001d21 by thread T128: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Atomic write of size 8 at 0x7dd000001d10 by thread T126: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x000000297d2b) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000297dca) #9 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x0000002965d3) #11 (libtablet.so+0x00000028cafd) #12 (libtablet.so+0x00000028936b) #13 (libtablet.so+0x0000001dcebd) #14 (libtablet.so+0x0000001dde5d) #15 (libtablet.so+0x0000001ddcaf) #16 (libtablet.so+0x0000002438f1) #17 (libtablet.so+0x00000023b2f5) #18 (libtablet.so+0x00000023d2c6) #19 (libtablet.so+0x00000023d255) #20 (libtablet.so+0x00000023d1ea) #21 (libconsensus.so+0x0000000befdd) #22 (libconsensus.so+0x0000000c5726) #23 (libconsensus.so+0x0000000c568e) #24 (libconsensus.so+0x0000000c5643) #25 (libconsensus.so+0x0000000c5401) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x000000226bcd) #28 (libkudu_util.so+0x000000224fcf) #29 (libkudu_util.so+0x00000022941e) #30 (libkudu_util.so+0x00000022937b) #31 (libkudu_util.so+0x000000229303) #32 (libkudu_util.so+0x000000229101) #33 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x00000021b55f) #35 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dd000001d11 by thread T128: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7dd000001d00 by thread T126: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000297dca) #6 (libtablet.so+0x000000297dca) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd000001d00 by thread T128: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7f4ccc24fd58 by thread T122: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f4ccc24fd59 by thread T124: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f4ccc207000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Atomic write of size 8 at 0x7f4ccc24fd50 by thread T122: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000297dca) #9 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x0000002965d3) #11 (libtablet.so+0x00000028cafd) #12 (libtablet.so+0x00000028936b) #13 (libtablet.so+0x0000001dcebd) #14 (libtablet.so+0x0000001dde5d) #15 (libtablet.so+0x0000001ddcaf) #16 (libtablet.so+0x0000002438f1) #17 (libtablet.so+0x00000023b2f5) #18 (libtablet.so+0x00000023d2c6) #19 (libtablet.so+0x00000023d255) #20 (libtablet.so+0x00000023d1ea) #21 (libconsensus.so+0x0000000befdd) #22 (libconsensus.so+0x0000000c5726) #23 (libconsensus.so+0x0000000c568e) #24 (libconsensus.so+0x0000000c5643) #25 (libconsensus.so+0x0000000c5401) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x000000226bcd) #28 (libkudu_util.so+0x000000224fcf) #29 (libkudu_util.so+0x00000022941e) #30 (libkudu_util.so+0x00000022937b) #31 (libkudu_util.so+0x000000229303) #32 (libkudu_util.so+0x000000229101) #33 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x00000021b55f) #35 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7f4ccc24fd51 by thread T124: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f4ccc207000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7f82605b5380 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f82605b5381 by thread T126: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f826053b000 allocated by thread T128: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=30353, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7f82605ac090 by thread T125: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f82605ac090 by thread T127: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f826053b000 allocated by thread T128: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7f825bd2b8d0 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7f825bd2b8d1 by thread T128: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f825bd07000 allocated by thread T127: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=30025) Write of size 8 at 0x7f825bd2b948 by thread T125: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f825bd2b948 by thread T128: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f825bd07000 allocated by thread T127: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=30352, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=30355, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=30354, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=30142) Write of size 8 at 0x7f4ccc283560 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7f4ccc283562 by thread T124: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f4ccc207000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=30357, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=30361, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=30356, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== I0628 23:55:07.253141 30337 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:55:07.257004 30338 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:55:07.283412 30337 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 I0628 23:55:07.293563 30338 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 W0628 23:55:07.354717 30136 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:36382 forcing change of non-runtime-safe flag fault_crash_on_handle_rb_fetch_data I0628 23:55:07.355013 30136 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:36382 changed flags via RPC: fault_crash_on_handle_rb_fetch_data from '0' to '1.0' I0628 23:55:07.356317 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:35960 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=37992 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:48620 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0 --enable_leader_failure_detection=false --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:07.470940 30372 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:07.471563 30372 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:07.486917 30372 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:07.489693 30372 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 139851 I0628 23:55:07.504695 30372 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0 uuid: "a9e0262858964a4387d95eaa285fa762" format_stamp: "Formatted at 2016-06-28 23:54:56 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:07.540730 30372 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:07.638383 30372 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:35960 I0628 23:55:07.639924 30372 webserver.cc:122] Starting webserver on 127.106.77.0:37992 I0628 23:55:07.640015 30372 webserver.cc:133] Document root disabled I0628 23:55:07.641536 30372 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:37992/ I0628 23:55:07.642928 30372 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:07.644940 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 30372 I0628 23:55:07.645200 30372 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:07.649049 30096 consensus_peers.cc:351] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Closing peer: c3a57434d605412c82946ec0f6f80b2a I0628 23:55:07.649847 30096 consensus_queue.cc:145] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 1.444, Majority replicated op: 1.444, Committed index: 1.444, Last appended: 1.444, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } I0628 23:55:07.655465 30486 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:48620 I0628 23:55:07.655930 30486 heartbeater.cc:319] Registering TS with master... I0628 23:55:07.656214 30486 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:07.657941 30285 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" instance_seqno: 1467158107618857 } with Master I0628 23:55:07.659984 30212 raft_consensus.cc:869] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 1 FOLLOWER]: Refusing update from remote peer 8bea06ed38f24dd3b85c977539f7d5f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 444. Preceding OpId from leader: term: 1 index: 445. (index mismatch) I0628 23:55:07.661485 30346 consensus_queue.cc:578] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [LEADER]: Connected to new peer: Peer: c3a57434d605412c82946ec0f6f80b2a, Is new: false, Last received: 1.444, Next index: 445, Last known committed idx: 444, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:55:07.665185 30339 consensus_peers.cc:296] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Sending request to remotely bootstrap I0628 23:55:07.666577 30443 ts_tablet_manager.cc:412] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Initiating remote bootstrap from Peer 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602) I0628 23:55:07.666822 30443 remote_bootstrap_client.cc:164] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Beginning remote bootstrap session from remote peer at address 127.106.77.1:40602 I0628 23:55:07.666870 30348 raft_consensus_state.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 LEADER]: Committing config change with OpId 1.445. Old config: { opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } }. New config: { opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } } I0628 23:55:07.672207 30116 remote_bootstrap_service.cc:123] Beginning new remote bootstrap session on tablet 81bb583ab6264dd3bed571d157bdc78e from peer a9e0262858964a4387d95eaa285fa762 at {real_user=slave, eff_user=} at 127.106.77.0:42826: session id = a9e0262858964a4387d95eaa285fa762-81bb583ab6264dd3bed571d157bdc78e I0628 23:55:07.675225 30285 catalog_manager.cc:1673] Tablet: 81bb583ab6264dd3bed571d157bdc78e reported consensus state change. New consensus state: current_term: 1 leader_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" config { opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } } I0628 23:55:07.676041 30116 remote_bootstrap_session.cc:142] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Remote bootstrap: opened 0 blocks and 2 log segments I0628 23:55:07.691787 30443 ts_tablet_manager.cc:760] Registered tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:55:07.692190 30443 remote_bootstrap_client.cc:387] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Starting download of 0 data blocks... I0628 23:55:07.692493 30443 remote_bootstrap_client.cc:352] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Starting download of 2 WAL segments... I0628 23:55:07.692667 30443 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: RemoteBootstrap: Downloading WAL segment with seq. number 1 (1/2) W0628 23:55:07.694008 30116 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:55:07.694104 30116 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:55:07.694278 30116 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_on_handle_rb_fetch_data *** Check failure stack trace: *** @ 0x7f82aba47304 google::LogMessage::SendToLog() at ??:0 @ 0x7f82aba47800 google::LogMessage::Flush() at ??:0 @ 0x7f82aba4b432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f82ac712e42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7f82b35cee6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f82b35cd1fe kudu::tserver::RemoteBootstrapServiceImpl::FetchData() at ??:0 @ 0x7f82b2e70745 kudu::tserver::RemoteBootstrapServiceIf::RemoteBootstrapServiceIf()::$_2::operator()() at ??:0 @ 0x7f82b2e7055f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f82b0ce3d49 std::function<>::operator()() at ??:0 @ 0x7f82b0ce3949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7f82b0ce569e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f82b0ce7477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f82b0ce73db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f82b0ce7384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f82b0ce71aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f82b0c8ee72 boost::function0<>::operator()() at ??:0 @ 0x7f82ac77d560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7f82ae1bc182 start_thread at ??:0 @ 0x7f82a9d4b47d clone at ??:0 @ (nil) (unknown) W0628 23:55:07.799329 30443 ts_tablet_manager.cc:989] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap: Unable to fetch data from remote peer 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Network error: Unable to download WAL segment with seq. number 1: Unable to fetch data from remote: Recv() got EOF from remote (error 108) I0628 23:55:07.799543 30443 ts_tablet_manager.cc:992] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Tombstoning tablet after failed remote bootstrap I0628 23:55:07.799619 30443 ts_tablet_manager.cc:955] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:55:07.803627 30443 ts_tablet_manager.cc:965] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Tablet deleted. Last logged OpId: 0.0 W0628 23:55:07.805701 30443 remote_bootstrap_client.cc:112] Unable to close remote bootstrap session: Network error: Failure ending remote bootstrap session: Client connection negotiation failed: client connection to 127.106.77.1:40602: connect: Connection refused (error 111) W0628 23:55:07.806596 30377 connection.cc:372] Connection torn down before Call kudu.consensus.ConsensusService.StartRemoteBootstrap from 127.106.77.1:50727 (request call id 2) could send its response I0628 23:55:07.825055 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 30142 I0628 23:55:07.854446 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:40602 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_port=48507 --webserver_interface=127.106.77.1 --tserver_master_addrs=127.0.0.1:48620 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1 --enable_leader_failure_detection=false --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:07.970717 30515 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:07.971382 30515 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:07.984657 30515 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:07.987363 30515 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 139851 I0628 23:55:07.999840 30515 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1 uuid: "8bea06ed38f24dd3b85c977539f7d5f5" format_stamp: "Formatted at 2016-06-28 23:54:56 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:08.034116 30515 ts_tablet_manager.cc:604] Loading metadata for tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:55:08.039973 30515 ts_tablet_manager.cc:760] Registered tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:55:08.041878 30525 ts_tablet_manager.cc:628] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Bootstrapping tablet I0628 23:55:08.046140 30525 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Bootstrap starting. I0628 23:55:08.050427 30525 tablet_bootstrap.cc:553] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:08.051045 30525 tablet_bootstrap.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 I0628 23:55:08.051313 30525 tablet_bootstrap.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000001 I0628 23:55:08.051436 30525 tablet_bootstrap.cc:617] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery in preparation for log replay W0628 23:55:08.052666 30525 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002: Not found: Footer not found. Footer magic doesn't match I0628 23:55:08.052858 30525 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:08.060473 30515 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:08.092396 30525 log_util.cc:569] Scanning /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 for valid entry headers following offset 8046... I0628 23:55:08.104470 30525 log_util.cc:606] Found no log entry headers I0628 23:55:08.105955 30525 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 8034 for log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002: Prior entries: [off=7294 COMMIT (1.444)] [off=7626 COMMIT (1.441)] [off=8007 REPLICATE (1.445)] [off=8034 COMMIT (1.445)] I0628 23:55:08.106359 30525 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 (valid entries through byte offset 8034) I0628 23:55:08.108055 30525 tablet.cc:854] T 81bb583ab6264dd3bed571d157bdc78e Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0628 23:55:08.113073 30525 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:08.196620 30515 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:40602 I0628 23:55:08.198097 30515 webserver.cc:122] Starting webserver on 127.106.77.1:48507 I0628 23:55:08.198186 30515 webserver.cc:133] Document root disabled I0628 23:55:08.208617 30515 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:48507/ I0628 23:55:08.209911 30515 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:55:08.213762 30515 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:08.213742 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 30515 I0628 23:55:08.214825 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:48322 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_port=46741 --webserver_interface=127.106.77.2 --tserver_master_addrs=127.0.0.1:48620 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2 --enable_leader_failure_detection=false --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:08.241899 30632 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:48620 I0628 23:55:08.242331 30632 heartbeater.cc:319] Registering TS with master... I0628 23:55:08.242604 30632 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:08.244905 30285 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" instance_seqno: 1467158108153687 } with Master I0628 23:55:08.336496 30634 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:08.336967 30634 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:08.360363 30634 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:08.363008 30634 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 140351 I0628 23:55:08.383307 30634 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2 uuid: "c3a57434d605412c82946ec0f6f80b2a" format_stamp: "Formatted at 2016-06-28 23:54:57 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:08.458458 30634 ts_tablet_manager.cc:604] Loading metadata for tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:55:08.466346 30634 ts_tablet_manager.cc:760] Registered tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:55:08.485957 30649 ts_tablet_manager.cc:628] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Bootstrapping tablet I0628 23:55:08.486886 30634 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:08.491492 30649 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Bootstrap starting. I0628 23:55:08.507436 30649 tablet_bootstrap.cc:553] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Time spent opening tablet: real 0.002s user 0.000s sys 0.000s I0628 23:55:08.510116 30649 tablet_bootstrap.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 I0628 23:55:08.511939 30649 tablet_bootstrap.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000001 I0628 23:55:08.514178 30649 tablet_bootstrap.cc:617] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery in preparation for log replay W0628 23:55:08.516247 30649 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002: Not found: Footer not found. Footer magic doesn't match I0628 23:55:08.516544 30649 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:08.569735 30649 log_util.cc:569] Scanning /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 for valid entry headers following offset 8683... I0628 23:55:08.578604 30649 log_util.cc:606] Found no log entry headers I0628 23:55:08.579744 30649 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 8671 for log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002: Prior entries: [off=7626 COMMIT (1.442)] [off=7958 COMMIT (1.444)] [off=8290 COMMIT (1.441)] [off=8671 REPLICATE (1.445)] I0628 23:55:08.580080 30649 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 (valid entries through byte offset 8671) I0628 23:55:08.581681 30649 tablet.cc:854] T 81bb583ab6264dd3bed571d157bdc78e Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0628 23:55:08.586258 30649 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:08.858700 30634 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:48322 I0628 23:55:08.860180 30634 webserver.cc:122] Starting webserver on 127.106.77.2:46741 I0628 23:55:08.860388 30634 webserver.cc:133] Document root disabled I0628 23:55:08.872390 30634 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:46741/ I0628 23:55:08.873793 30634 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:55:08.880663 30634 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:08.880986 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 30634 I0628 23:55:08.914249 30756 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:48620 I0628 23:55:08.914665 30756 heartbeater.cc:319] Registering TS with master... I0628 23:55:08.914949 30756 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:08.916776 30285 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" instance_seqno: 1467158108825495 } with Master I0628 23:55:13.414813 30525 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:55:13.416471 30525 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Bootstrap replayed 1/2 log segments. Stats: ops{read=441 overwritten=0 applied=440 ignored=0} inserts{seen=21950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0628 23:55:13.417875 30525 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 I0628 23:55:13.484220 30525 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Bootstrap replayed 2/2 log segments. Stats: ops{read=445 overwritten=0 applied=445 ignored=0} inserts{seen=22150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:55:13.484573 30525 tablet_bootstrap.cc:1030] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: ReplayState: Previous OpId: 1.445, Committed OpId: 1.445, Pending Replicates: 0, Pending Commits: 0 I0628 23:55:13.490082 30525 tablet_bootstrap.cc:646] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery I0628 23:55:13.490447 30525 tablet_bootstrap.cc:649] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158113490313 I0628 23:55:13.490742 30525 tablet_bootstrap.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158113490313 I0628 23:55:13.491652 30525 tablet_bootstrap.cc:662] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-1/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158113490313 I0628 23:55:13.491986 30525 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Bootstrap complete. I0628 23:55:13.492954 30525 ts_tablet_manager.cc:633] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Time spent bootstrapping tablet: real 5.451s user 5.344s sys 0.056s I0628 23:55:13.499785 30525 raft_consensus.cc:269] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } I0628 23:55:13.501412 30525 raft_consensus.cc:499] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bea06ed38f24dd3b85c977539f7d5f5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 445 Committed: term: 1 index: 445} I0628 23:55:13.501991 30525 consensus_queue.cc:162] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.445, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:13.505043 30525 ts_tablet_manager.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s I0628 23:55:14.182651 30649 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:55:14.184521 30649 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Bootstrap replayed 1/2 log segments. Stats: ops{read=441 overwritten=0 applied=438 ignored=0} inserts{seen=21850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates I0628 23:55:14.195935 30649 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 I0628 23:55:14.266217 30649 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Bootstrap replayed 2/2 log segments. Stats: ops{read=445 overwritten=0 applied=444 ignored=0} inserts{seen=22150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0628 23:55:14.267779 30649 tablet_bootstrap.cc:1030] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: ReplayState: Previous OpId: 1.445, Committed OpId: 1.444, Pending Replicates: 1, Pending Commits: 0 I0628 23:55:14.267902 30649 tablet_bootstrap.cc:1030] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Dumping REPLICATES: I0628 23:55:14.268005 30649 tablet_bootstrap.cc:1030] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: type: REPLICATE replicate { id { term: 1 index: 445 } timestamp: 6009479608924172288 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "81bb583ab6264dd3bed571d157bdc78e" old_config { opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } } new_config { peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } } } } I0628 23:55:14.271895 30649 tablet_bootstrap.cc:646] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery I0628 23:55:14.272052 30649 tablet_bootstrap.cc:649] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158114271995 I0628 23:55:14.272277 30649 tablet_bootstrap.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158114271995 I0628 23:55:14.273021 30649 tablet_bootstrap.cc:662] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-2/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158114271995 I0628 23:55:14.273267 30649 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Bootstrap complete. I0628 23:55:14.273970 30649 ts_tablet_manager.cc:633] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Time spent bootstrapping tablet: real 5.786s user 5.320s sys 0.084s I0628 23:55:14.278717 30649 raft_consensus.cc:269] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } I0628 23:55:14.280285 30649 raft_consensus.cc:499] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a57434d605412c82946ec0f6f80b2a, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 445 Committed: term: 1 index: 444} I0628 23:55:14.280668 30649 consensus_queue.cc:162] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.445, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:14.283871 30649 ts_tablet_manager.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s I0628 23:55:14.284627 30589 raft_consensus.cc:370] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:14.284818 30589 raft_consensus.cc:2019] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 1 FOLLOWER]: Advancing to term 2 I0628 23:55:14.287104 30589 raft_consensus.cc:386] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 2 FOLLOWER]: Starting election with config: opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } I0628 23:55:14.291234 30589 leader_election.cc:223] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [CANDIDATE]: Term 2 election: Requesting vote from peer c3a57434d605412c82946ec0f6f80b2a I0628 23:55:14.291719 30589 leader_election.cc:223] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [CANDIDATE]: Term 2 election: Requesting vote from peer a9e0262858964a4387d95eaa285fa762 W0628 23:55:14.303766 30520 leader_election.cc:281] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a9e0262858964a4387d95eaa285fa762: Not found: Tablet not found I0628 23:55:14.305227 30713 raft_consensus.cc:2019] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 1 FOLLOWER]: Advancing to term 2 I0628 23:55:14.308876 30713 raft_consensus.cc:1677] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bea06ed38f24dd3b85c977539f7d5f5 in term 2. I0628 23:55:14.309587 30520 leader_election.cc:347] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [CANDIDATE]: Term 2 election: Vote granted by peer c3a57434d605412c82946ec0f6f80b2a I0628 23:55:14.309767 30520 leader_election.cc:248] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0628 23:55:14.311048 30781 raft_consensus.cc:1859] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 2 FOLLOWER]: Leader election won for term 2 I0628 23:55:14.312355 30781 raft_consensus.cc:464] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 2 LEADER]: Becoming Leader. State: Replica: 8bea06ed38f24dd3b85c977539f7d5f5, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 445 Committed: term: 1 index: 445} I0628 23:55:14.312827 30781 consensus_queue.cc:145] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 1.445, Committed index: 1.445, Last appended: 1.445, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } I0628 23:55:14.320024 30285 catalog_manager.cc:1673] Tablet: 81bb583ab6264dd3bed571d157bdc78e reported consensus state change. New consensus state: current_term: 2 leader_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" config { opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } } W0628 23:55:14.322834 30785 raft_consensus_state.cc:541] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 2 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 1 index: 445, New majority replicated is: term: 1 index: 445, Current term is: 2 I0628 23:55:14.819249 30713 raft_consensus.cc:869] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 2 FOLLOWER]: Refusing update from remote peer 8bea06ed38f24dd3b85c977539f7d5f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 445. Preceding OpId from leader: term: 2 index: 446. (index mismatch) I0628 23:55:14.820667 30781 consensus_peers.cc:296] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Sending request to remotely bootstrap I0628 23:55:14.821177 30786 consensus_queue.cc:578] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [LEADER]: Connected to new peer: Peer: c3a57434d605412c82946ec0f6f80b2a, Is new: false, Last received: 1.445, Next index: 446, Last known committed idx: 444, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:55:14.822115 30443 ts_tablet_manager.cc:412] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Initiating remote bootstrap from Peer 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602) I0628 23:55:14.822623 30443 remote_bootstrap_client.cc:164] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Beginning remote bootstrap session from remote peer at address 127.106.77.1:40602 I0628 23:55:14.827000 30713 raft_consensus_state.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a [term 2 FOLLOWER]: Committing config change with OpId 1.445. Old config: { opid_index: -1 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } }. New config: { opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } } I0628 23:55:14.830153 30609 remote_bootstrap_service.cc:123] Beginning new remote bootstrap session on tablet 81bb583ab6264dd3bed571d157bdc78e from peer a9e0262858964a4387d95eaa285fa762 at {real_user=slave, eff_user=} at 127.106.77.0:34868: session id = a9e0262858964a4387d95eaa285fa762-81bb583ab6264dd3bed571d157bdc78e I0628 23:55:14.833500 30609 remote_bootstrap_session.cc:142] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Remote bootstrap: opened 0 blocks and 2 log segments I0628 23:55:14.839754 30443 ts_tablet_manager.cc:760] Registered tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:55:14.840070 30443 remote_bootstrap_client.cc:387] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Starting download of 0 data blocks... I0628 23:55:14.840391 30443 remote_bootstrap_client.cc:352] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Starting download of 2 WAL segments... I0628 23:55:14.840596 30443 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: RemoteBootstrap: Downloading WAL segment with seq. number 1 (1/2) I0628 23:55:14.849869 30790 raft_consensus_state.cc:534] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 [term 2 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 1 index: 445 New committed index is: term: 2 index: 446 I0628 23:55:14.872025 30443 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: RemoteBootstrap: Downloading WAL segment with seq. number 2 (2/2) I0628 23:55:14.877550 30443 remote_bootstrap_client.cc:273] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Remote bootstrap complete. Replacing tablet superblock. I0628 23:55:14.877678 30443 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: RemoteBootstrap: Replacing tablet superblock I0628 23:55:14.881811 30794 ts_tablet_manager.cc:628] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrapping tablet I0628 23:55:14.882457 30609 remote_bootstrap_service.cc:242] Request end of remote bootstrap session a9e0262858964a4387d95eaa285fa762-81bb583ab6264dd3bed571d157bdc78e received from {real_user=slave, eff_user=} at 127.106.77.0:34868 I0628 23:55:14.882625 30609 remote_bootstrap_service.cc:323] Ending remote bootstrap session a9e0262858964a4387d95eaa285fa762-81bb583ab6264dd3bed571d157bdc78e on tablet 81bb583ab6264dd3bed571d157bdc78e with peer a9e0262858964a4387d95eaa285fa762 I0628 23:55:14.883942 30794 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrap starting. I0628 23:55:14.888182 30794 tablet_bootstrap.cc:553] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:55:14.888761 30794 tablet_bootstrap.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 I0628 23:55:14.889014 30794 tablet_bootstrap.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000001 I0628 23:55:14.889173 30794 tablet_bootstrap.cc:617] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery in preparation for log replay W0628 23:55:14.890369 30794 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002: Not found: Footer not found. Footer magic doesn't match I0628 23:55:14.890576 30794 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:14.893806 30794 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 (valid entries through byte offset 8073) I0628 23:55:14.894812 30794 tablet.cc:854] T 81bb583ab6264dd3bed571d157bdc78e Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0628 23:55:14.897333 30794 log.cc:350] Log is configured to *not* fsync() on all Append() calls INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:15.316601 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 1 times. W0628 23:55:15.659734 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0628 23:55:16.160401 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:16.586205 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected W0628 23:55:16.660214 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:17.161092 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:17.300277 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:17.661545 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:18.147919 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected W0628 23:55:18.161392 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:18.662328 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:19.106706 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected W0628 23:55:19.162309 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 9 times. INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:19.663769 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 10 times. W0628 23:55:20.163889 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 11 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:20.212990 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected W0628 23:55:20.664422 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 12 times. INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY I0628 23:55:21.061913 30794 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:55:21.072978 30794 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrap replayed 1/2 log segments. Stats: ops{read=441 overwritten=0 applied=440 ignored=0} inserts{seen=21950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0628 23:55:21.074271 30794 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 I0628 23:55:21.155814 30794 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrap replayed 2/2 log segments. Stats: ops{read=446 overwritten=0 applied=445 ignored=0} inserts{seen=22150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates W0628 23:55:21.165734 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 13 times. I0628 23:55:21.190120 30794 tablet_bootstrap.cc:1030] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: ReplayState: Previous OpId: 2.446, Committed OpId: 1.445, Pending Replicates: 1, Pending Commits: 0 I0628 23:55:21.190354 30794 tablet_bootstrap.cc:1030] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Dumping REPLICATES: I0628 23:55:21.190551 30794 tablet_bootstrap.cc:1030] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: type: REPLICATE replicate { id { term: 2 index: 446 } timestamp: 6009479636242034688 op_type: NO_OP noop_request { } } I0628 23:55:21.193514 30794 tablet_bootstrap.cc:646] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery I0628 23:55:21.193763 30794 tablet_bootstrap.cc:649] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158121193698 I0628 23:55:21.194059 30794 tablet_bootstrap.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158121193698 I0628 23:55:21.194830 30794 tablet_bootstrap.cc:662] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158121193698 I0628 23:55:21.195183 30794 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrap complete. I0628 23:55:21.196027 30794 ts_tablet_manager.cc:633] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Time spent bootstrapping tablet: real 6.314s user 5.968s sys 0.056s I0628 23:55:21.203263 30794 raft_consensus.cc:269] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } I0628 23:55:21.205314 30794 raft_consensus.cc:499] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e0262858964a4387d95eaa285fa762, State: 1, Role: FOLLOWER Watermarks: {Received: term: 2 index: 446 Committed: term: 1 index: 445} I0628 23:55:21.205860 30794 consensus_queue.cc:162] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 2.446, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:21.224498 30794 ts_tablet_manager.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Time spent starting tablet: real 0.028s user 0.004s sys 0.020s ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:21.502645 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 I0628 23:55:23.751534 30423 tablet_service.cc:653] Processing DeleteTablet for tablet 81bb583ab6264dd3bed571d157bdc78e with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:44874 I0628 23:55:23.751854 30423 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 81bb583ab6264dd3bed571d157bdc78e I0628 23:55:23.752187 30423 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(81bb583ab6264dd3bed571d157bdc78e) I0628 23:55:23.752413 30423 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(81bb583ab6264dd3bed571d157bdc78e) I0628 23:55:23.752599 30423 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(81bb583ab6264dd3bed571d157bdc78e) I0628 23:55:23.752840 30423 maintenance_manager.cc:180] Unregistered op FlushMRSOp(81bb583ab6264dd3bed571d157bdc78e) I0628 23:55:23.753012 30423 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(81bb583ab6264dd3bed571d157bdc78e) I0628 23:55:23.753188 30423 maintenance_manager.cc:180] Unregistered op LogGCOp(81bb583ab6264dd3bed571d157bdc78e) I0628 23:55:23.753340 30423 raft_consensus.cc:1495] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 [term 2 FOLLOWER]: Raft consensus shutting down. I0628 23:55:23.753696 30423 raft_consensus.cc:1511] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 [term 2 FOLLOWER]: Raft consensus is shut down! I0628 23:55:23.783758 30423 ts_tablet_manager.cc:955] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:55:23.788224 30423 ts_tablet_manager.cc:965] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Tablet deleted. Last logged OpId: 2.446 I0628 23:55:23.789242 30423 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Deleted tablet blocks from disk I0628 23:55:24.168053 30786 consensus_peers.cc:296] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Sending request to remotely bootstrap I0628 23:55:24.169658 30443 ts_tablet_manager.cc:412] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Initiating remote bootstrap from Peer 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602) I0628 23:55:24.172372 30443 remote_bootstrap_client.cc:164] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Beginning remote bootstrap session from remote peer at address 127.106.77.1:40602 I0628 23:55:24.173506 30609 remote_bootstrap_service.cc:123] Beginning new remote bootstrap session on tablet 81bb583ab6264dd3bed571d157bdc78e from peer a9e0262858964a4387d95eaa285fa762 at {real_user=slave, eff_user=} at 127.106.77.0:34868: session id = a9e0262858964a4387d95eaa285fa762-81bb583ab6264dd3bed571d157bdc78e I0628 23:55:24.176309 30609 remote_bootstrap_session.cc:142] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5: Remote bootstrap: opened 0 blocks and 2 log segments I0628 23:55:24.180515 30443 ts_tablet_manager.cc:760] Registered tablet 81bb583ab6264dd3bed571d157bdc78e I0628 23:55:24.180790 30443 remote_bootstrap_client.cc:387] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Starting download of 0 data blocks... I0628 23:55:24.181151 30443 remote_bootstrap_client.cc:352] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Starting download of 2 WAL segments... I0628 23:55:24.181330 30443 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: RemoteBootstrap: Downloading WAL segment with seq. number 1 (1/2) I0628 23:55:24.206612 30443 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: RemoteBootstrap: Downloading WAL segment with seq. number 2 (2/2) I0628 23:55:24.211839 30443 remote_bootstrap_client.cc:273] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Remote bootstrap client: Remote bootstrap complete. Replacing tablet superblock. I0628 23:55:24.212061 30443 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: RemoteBootstrap: Replacing tablet superblock I0628 23:55:24.215893 30879 ts_tablet_manager.cc:628] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrapping tablet I0628 23:55:24.216539 30609 remote_bootstrap_service.cc:242] Request end of remote bootstrap session a9e0262858964a4387d95eaa285fa762-81bb583ab6264dd3bed571d157bdc78e received from {real_user=slave, eff_user=} at 127.106.77.0:34868 I0628 23:55:24.216753 30609 remote_bootstrap_service.cc:323] Ending remote bootstrap session a9e0262858964a4387d95eaa285fa762-81bb583ab6264dd3bed571d157bdc78e on tablet 81bb583ab6264dd3bed571d157bdc78e with peer a9e0262858964a4387d95eaa285fa762 I0628 23:55:24.218289 30879 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrap starting. I0628 23:55:24.220988 30879 tablet_bootstrap.cc:553] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:24.221645 30879 tablet_bootstrap.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 I0628 23:55:24.221972 30879 tablet_bootstrap.cc:609] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000001 I0628 23:55:24.222131 30879 tablet_bootstrap.cc:617] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery in preparation for log replay W0628 23:55:24.222934 30879 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002: Not found: Footer not found. Footer magic doesn't match I0628 23:55:24.223111 30879 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:24.226879 30879 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery/wal-000000002 (valid entries through byte offset 8100) I0628 23:55:24.227594 30879 tablet.cc:854] T 81bb583ab6264dd3bed571d157bdc78e Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:24.668107 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 1 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:24.987141 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected INFO: Connected to the Master W0628 23:55:25.168359 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:25.668792 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:25.903738 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:26.169433 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0628 23:55:26.670084 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:26.899420 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected W0628 23:55:27.170027 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:27.670856 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:27.858778 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected W0628 23:55:28.171049 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY W0628 23:55:28.671658 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 9 times. ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Error: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0628 23:55:28.972892 27213 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Aborted: 1 errors were detected W0628 23:55:29.171638 30520 consensus_peers.cc:326] T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 -> Peer a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Couldn't send request to peer a9e0262858964a4387d95eaa285fa762 for tablet 81bb583ab6264dd3bed571d157bdc78e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 10 times. I0628 23:55:29.373352 30879 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:55:29.374548 30879 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrap replayed 1/2 log segments. Stats: ops{read=441 overwritten=0 applied=440 ignored=0} inserts{seen=21950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0628 23:55:29.375632 30879 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e/wal-000000002 I0628 23:55:29.429011 30879 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrap replayed 2/2 log segments. Stats: ops{read=446 overwritten=0 applied=446 ignored=0} inserts{seen=22150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:55:29.429203 30879 tablet_bootstrap.cc:1030] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: ReplayState: Previous OpId: 2.446, Committed OpId: 2.446, Pending Replicates: 0, Pending Commits: 0 I0628 23:55:29.431098 30879 tablet_bootstrap.cc:646] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery I0628 23:55:29.431304 30879 tablet_bootstrap.cc:649] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158129431250 I0628 23:55:29.431478 30879 tablet_bootstrap.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158129431250 I0628 23:55:29.432005 30879 tablet_bootstrap.cc:662] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails.1467158082783852-27213/minicluster-data/ts-0/wals/81bb583ab6264dd3bed571d157bdc78e.recovery-1467158129431250 I0628 23:55:29.432306 30879 tablet_bootstrap.cc:393] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Bootstrap complete. I0628 23:55:29.432817 30879 ts_tablet_manager.cc:633] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Time spent bootstrapping tablet: real 5.217s user 5.152s sys 0.060s I0628 23:55:29.435849 30879 raft_consensus.cc:269] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 445 peers { permanent_uuid: "c3a57434d605412c82946ec0f6f80b2a" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48322 } } peers { permanent_uuid: "8bea06ed38f24dd3b85c977539f7d5f5" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 40602 } } peers { permanent_uuid: "a9e0262858964a4387d95eaa285fa762" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35960 } } I0628 23:55:29.436498 30879 raft_consensus.cc:499] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e0262858964a4387d95eaa285fa762, State: 1, Role: FOLLOWER Watermarks: {Received: term: 2 index: 446 Committed: term: 2 index: 446} I0628 23:55:29.436904 30879 consensus_queue.cc:162] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 2.446, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:29.438953 30879 ts_tablet_manager.cc:659] T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s INFO: Connected to the Master INFO: Connected to all 3 Tablet Servers WARNING: Tablet 81bb583ab6264dd3bed571d157bdc78e of table 'test-workload' has 3 instead of 2 replicas INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY ----------------------- test-workload ----------------------- T 81bb583ab6264dd3bed571d157bdc78e P a9e0262858964a4387d95eaa285fa762 (127.106.77.0:35960): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P c3a57434d605412c82946ec0f6f80b2a (127.106.77.2:48322): Checksum: 47299235369098 T 81bb583ab6264dd3bed571d157bdc78e P 8bea06ed38f24dd3b85c977539f7d5f5 (127.106.77.1:40602): Checksum: 47299235369098 I0628 23:55:30.460386 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 30266 I0628 23:55:30.486788 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 30372 I0628 23:55:30.521714 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 30515 I0628 23:55:30.556313 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 30634 [ OK ] DeleteTableTest.TestAutoTombstoneAfterRemoteBootstrapRemoteFails (34807 ms) [ RUN ] DeleteTableTest.TestMergeConsensusMetadata I0628 23:55:30.590682 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/master --catalog_manager_wait_for_new_tablets_to_elect_leader=false --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:30.703812 30956 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:30.704279 30956 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:30.716707 30956 master_main.cc:58] Initializing master server... I0628 23:55:30.718869 30956 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 151351 I0628 23:55:30.725805 30956 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:55:30.726001 30956 server_base.cc:165] Creating new FS layout I0628 23:55:30.739128 30956 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/master/instance: uuid: "b53a2beb89fb4a26b19af3ba50b2228b" format_stamp: "Formatted at 2016-06-28 23:55:30 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:30.751901 30956 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/master uuid: "b53a2beb89fb4a26b19af3ba50b2228b" format_stamp: "Formatted at 2016-06-28 23:55:30 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:30.785532 30956 master_main.cc:61] Starting Master server... I0628 23:55:30.820389 30956 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:43578 I0628 23:55:30.822013 30956 webserver.cc:122] Starting webserver on localhost:0 I0628 23:55:30.822134 30956 webserver.cc:133] Document root disabled I0628 23:55:30.822906 30956 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:55:30.824172 30956 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:47554/ I0628 23:55:30.825453 30956 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:55:30.828055 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 30956 I0628 23:55:30.828888 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43578 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0 --enable_leader_failure_detection=false --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:30.850769 30999 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b: Bootstrap starting. I0628 23:55:30.854594 30999 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:30.855234 30999 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b: No blocks or log segments found. Creating new log. I0628 23:55:30.857280 30999 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:30.862309 30999 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b: No bootstrap required, opened a new log I0628 23:55:30.869086 30999 raft_consensus.cc:269] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" member_type: VOTER } I0628 23:55:30.870645 30999 raft_consensus.cc:301] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:30.870997 30999 raft_consensus.cc:499] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53a2beb89fb4a26b19af3ba50b2228b, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:30.871618 30999 consensus_queue.cc:162] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:30.872089 30999 raft_consensus.cc:314] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:55:30.872386 30999 raft_consensus.cc:370] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:30.872586 30999 raft_consensus.cc:2019] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:30.875164 30999 raft_consensus.cc:1970] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.541s I0628 23:55:30.875419 30999 raft_consensus.cc:386] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" member_type: VOTER } I0628 23:55:30.878322 30999 leader_election.cc:248] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:30.880277 31004 raft_consensus.cc:1970] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.563s I0628 23:55:30.880524 31004 raft_consensus.cc:1859] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:30.883186 31004 raft_consensus.cc:464] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 1 LEADER]: Becoming Leader. State: Replica: b53a2beb89fb4a26b19af3ba50b2228b, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:30.884114 31004 consensus_queue.cc:145] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" member_type: VOTER } I0628 23:55:30.904640 31005 sys_catalog.cc:235] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" config { opid_index: -1 peers { permanent_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" member_type: VOTER } } I0628 23:55:30.905030 30999 sys_catalog.cc:235] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" config { opid_index: -1 peers { permanent_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" member_type: VOTER } } I0628 23:55:30.905757 31005 sys_catalog.cc:238] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [sys.catalog]: This master's current role is: LEADER I0628 23:55:30.905886 30999 sys_catalog.cc:238] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [sys.catalog]: This master's current role is: LEADER I0628 23:55:30.909020 31008 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:55:30.911005 31009 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:55:30.912029 30999 sys_catalog.cc:311] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [sys.catalog]: configured and running, proceeding with master startup. I0628 23:55:30.906749 31006 sys_catalog.cc:235] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53a2beb89fb4a26b19af3ba50b2228b. Latest consensus state: current_term: 1 leader_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" config { opid_index: -1 peers { permanent_uuid: "b53a2beb89fb4a26b19af3ba50b2228b" member_type: VOTER } } I0628 23:55:30.912364 31006 sys_catalog.cc:238] T 00000000000000000000000000000000 P b53a2beb89fb4a26b19af3ba50b2228b [sys.catalog]: This master's current role is: LEADER I0628 23:55:30.923316 30956 master_main.cc:64] Master server successfully started. I0628 23:55:30.959897 31000 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:30.960391 31000 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:30.978878 31000 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:30.981945 31000 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 151351 I0628 23:55:30.988306 31000 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:55:30.988539 31000 server_base.cc:165] Creating new FS layout I0628 23:55:30.997550 31000 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "13048edf8677461fad00fc37acf1aeb7" format_stamp: "Formatted at 2016-06-28 23:55:30 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:31.014730 31000 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0 uuid: "13048edf8677461fad00fc37acf1aeb7" format_stamp: "Formatted at 2016-06-28 23:55:30 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:31.091428 31000 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:31.435732 31000 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:57634 I0628 23:55:31.437160 31000 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:55:31.437270 31000 webserver.cc:133] Document root disabled I0628 23:55:31.438993 31000 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:53688/ I0628 23:55:31.440495 31000 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:31.441381 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 31000 I0628 23:55:31.442546 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43578 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --enable_leader_failure_detection=false --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:31.453232 31000 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:31.470464 31124 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43578 I0628 23:55:31.470907 31124 heartbeater.cc:319] Registering TS with master... I0628 23:55:31.471151 31124 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:31.472980 30976 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" instance_seqno: 1467158131398418 } with Master I0628 23:55:31.584919 31127 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:31.585521 31127 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:31.600213 31127 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:31.603085 31127 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 151851 I0628 23:55:31.609308 31127 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:55:31.609561 31127 server_base.cc:165] Creating new FS layout I0628 23:55:31.618373 31127 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "40e545c6b6c540f5b91a210d27d04982" format_stamp: "Formatted at 2016-06-28 23:55:31 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:31.626669 31127 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 uuid: "40e545c6b6c540f5b91a210d27d04982" format_stamp: "Formatted at 2016-06-28 23:55:31 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:31.662695 31127 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:31.755040 31127 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:38052 I0628 23:55:31.756556 31127 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:55:31.756647 31127 webserver.cc:133] Document root disabled I0628 23:55:31.758296 31127 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:43953/ I0628 23:55:31.759580 31127 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:55:31.762058 31127 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:31.763742 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 31127 I0628 23:55:31.764598 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43578 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --enable_leader_failure_detection=false --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:31.771595 31242 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43578 I0628 23:55:31.771925 31242 heartbeater.cc:319] Registering TS with master... I0628 23:55:31.772153 31242 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:31.773318 30976 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" instance_seqno: 1467158131736148 } with Master I0628 23:55:31.882704 31245 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:31.883159 31245 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:31.897150 31245 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:31.899907 31245 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 151851 I0628 23:55:31.905735 31245 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:55:31.905927 31245 server_base.cc:165] Creating new FS layout I0628 23:55:31.913955 31245 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "f0f910d23820467fa1cca6dff8ad5c0c" format_stamp: "Formatted at 2016-06-28 23:55:31 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:31.921618 31245 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 uuid: "f0f910d23820467fa1cca6dff8ad5c0c" format_stamp: "Formatted at 2016-06-28 23:55:31 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:31.956967 31245 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:32.049810 31245 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:35961 I0628 23:55:32.051136 31245 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:55:32.051223 31245 webserver.cc:133] Document root disabled I0628 23:55:32.052587 31245 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:43897/ I0628 23:55:32.054033 31245 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:55:32.056458 31245 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:32.064167 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 31245 I0628 23:55:32.064729 31359 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43578 I0628 23:55:32.065218 31359 heartbeater.cc:319] Registering TS with master... I0628 23:55:32.065439 31359 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:32.066817 30976 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" instance_seqno: 1467158132030776 } with Master I0628 23:55:32.071575 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:55:32.122683 30976 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:41294: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:55:32.170733 31061 tablet_service.cc:612] Processing CreateTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 (table=test-workload [id=962c80acd3a0462493761f77658afdaf]), partition=range: [(), ()) I0628 23:55:32.172897 31296 tablet_service.cc:612] Processing CreateTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 (table=test-workload [id=962c80acd3a0462493761f77658afdaf]), partition=range: [(), ()) I0628 23:55:32.173362 31179 tablet_service.cc:612] Processing CreateTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 (table=test-workload [id=962c80acd3a0462493761f77658afdaf]), partition=range: [(), ()) I0628 23:55:32.191185 31061 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:32.193838 31179 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:32.195543 31379 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Bootstrapping tablet I0628 23:55:32.199220 31379 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Bootstrap starting. I0628 23:55:32.201308 31296 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:32.202620 31380 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrapping tablet I0628 23:55:32.205426 31380 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrap starting. I0628 23:55:32.207942 31379 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:32.209014 31382 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrapping tablet I0628 23:55:32.209913 31379 tablet_bootstrap.cc:493] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: No blocks or log segments found. Creating new log. I0628 23:55:32.211236 31380 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:32.211814 31380 tablet_bootstrap.cc:493] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: No blocks or log segments found. Creating new log. I0628 23:55:32.213327 31379 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:32.213620 31380 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:32.216064 31382 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrap starting. I0628 23:55:32.218505 31380 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: No bootstrap required, opened a new log I0628 23:55:32.219003 31380 ts_tablet_manager.cc:633] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s I0628 23:55:32.220655 31382 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:55:32.221365 31382 tablet_bootstrap.cc:493] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: No blocks or log segments found. Creating new log. I0628 23:55:32.223211 31382 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:32.226686 31380 raft_consensus.cc:269] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:32.228060 31379 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: No bootstrap required, opened a new log I0628 23:55:32.228466 31382 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: No bootstrap required, opened a new log I0628 23:55:32.228165 31380 raft_consensus.cc:499] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e545c6b6c540f5b91a210d27d04982, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:32.228580 31379 ts_tablet_manager.cc:633] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.000s I0628 23:55:32.228831 31382 ts_tablet_manager.cc:633] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s I0628 23:55:32.228775 31380 consensus_queue.cc:162] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:32.233490 31382 raft_consensus.cc:269] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:32.234644 31382 raft_consensus.cc:499] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f910d23820467fa1cca6dff8ad5c0c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:32.235155 31382 consensus_queue.cc:162] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:32.236898 31379 raft_consensus.cc:269] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:32.238664 31379 raft_consensus.cc:499] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13048edf8677461fad00fc37acf1aeb7, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:32.239411 31379 consensus_queue.cc:162] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:32.240892 31380 ts_tablet_manager.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s I0628 23:55:32.242853 31382 ts_tablet_manager.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s I0628 23:55:32.243226 31379 ts_tablet_manager.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s I0628 23:55:32.273396 31199 raft_consensus.cc:370] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:32.273582 31199 raft_consensus.cc:2019] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:32.275616 31199 raft_consensus.cc:386] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:32.278843 31199 leader_election.cc:223] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 1 election: Requesting vote from peer f0f910d23820467fa1cca6dff8ad5c0c I0628 23:55:32.279419 31199 leader_election.cc:223] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 1 election: Requesting vote from peer 13048edf8677461fad00fc37acf1aeb7 I0628 23:55:32.287544 31316 raft_consensus.cc:2019] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:32.288578 31081 raft_consensus.cc:2019] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:32.292933 31316 raft_consensus.cc:1677] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e545c6b6c540f5b91a210d27d04982 in term 1. I0628 23:55:32.294136 31081 raft_consensus.cc:1677] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e545c6b6c540f5b91a210d27d04982 in term 1. I0628 23:55:32.294356 31136 leader_election.cc:347] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 1 election: Vote granted by peer f0f910d23820467fa1cca6dff8ad5c0c I0628 23:55:32.294626 31136 leader_election.cc:248] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:32.295076 31135 leader_election.cc:347] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 1 election: Vote granted by peer 13048edf8677461fad00fc37acf1aeb7 I0628 23:55:32.295367 31389 raft_consensus.cc:1859] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:32.297601 31389 raft_consensus.cc:464] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 LEADER]: Becoming Leader. State: Replica: 40e545c6b6c540f5b91a210d27d04982, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:32.298362 31389 consensus_queue.cc:145] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } W0628 23:55:32.343847 31400 raft_consensus_state.cc:541] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:55:32.345842 30972 catalog_manager.cc:1673] Tablet: de15e82351f348c0b54b3c21a2cc9c78 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "40e545c6b6c540f5b91a210d27d04982" config { opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } } I0628 23:55:32.443284 31316 raft_consensus.cc:869] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 1 FOLLOWER]: Refusing update from remote peer 40e545c6b6c540f5b91a210d27d04982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:55:32.443284 31081 raft_consensus.cc:869] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 1 FOLLOWER]: Refusing update from remote peer 40e545c6b6c540f5b91a210d27d04982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:55:32.445103 31397 consensus_queue.cc:578] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [LEADER]: Connected to new peer: Peer: 13048edf8677461fad00fc37acf1aeb7, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:55:32.445507 31389 consensus_queue.cc:578] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [LEADER]: Connected to new peer: Peer: f0f910d23820467fa1cca6dff8ad5c0c, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:55:32.492518 31400 raft_consensus_state.cc:534] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 2 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==31000==WARNING: Can't read from symbolizer at fd 27 ==31127==WARNING: Can't read from symbolizer at fd 29 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==31127==WARNING: Can't read from symbolizer at fd 29 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==31000==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==31127==WARNING: Can't write to symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==31000==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==31127==WARNING: Can't read from symbolizer at fd 29 ==31127==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=31127) Read of size 8 at 0x7d500003fd38 by thread T127: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500003fd38 by thread T128: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=31406, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=31408, running) created by thread T124 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=31380, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==31000==WARNING: Can't read from symbolizer at fd 27 I0628 23:55:32.725461 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 31127 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==31000==WARNING: external symbolizer didn't start up correctly! ==31000==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 4 at 0x7d700006f8f0 by thread T122: #0 (libtablet.so+0x000000299980) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T123: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299980) ================== ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 8 at 0x7d700006f8c0 by thread T122: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8c0 by thread T123: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) ================== ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 8 at 0x7d500003fd38 by thread T124: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500003fd38 by thread T123: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=31379, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=31000) Read of size 8 at 0x7d500003fd38 by thread T122: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500003fd38 by thread T123: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=31379, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 8 at 0x7d900004f960 by thread T124: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900004f960 by thread T123: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 8 at 0x7d900004f950 by thread T124: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900004f951 by thread T123: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) I0628 23:55:32.749672 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 31245 #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 1 at 0x7d900004f949 by thread T124: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900004f948 by thread T123: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb826) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 8 at 0x7d900004f9d0 by thread T124: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900004f9d0 by thread T123: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=31000) Atomic write of size 4 at 0x7d900004fa28 by thread T124: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900004fa28 by thread T123: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== I0628 23:55:32.770424 31081 raft_consensus.cc:366] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 1 FOLLOWER]: Failure of leader 40e545c6b6c540f5b91a210d27d04982 detected. Triggering leader election I0628 23:55:32.770622 31081 raft_consensus.cc:2019] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 1 FOLLOWER]: Advancing to term 2 I0628 23:55:32.772917 31081 raft_consensus.cc:386] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 8 at 0x7d900004f970 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900004f971 by thread T124: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== I0628 23:55:32.778010 31081 leader_election.cc:223] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [CANDIDATE]: Term 2 election: Requesting vote from peer f0f910d23820467fa1cca6dff8ad5c0c I0628 23:55:32.778549 31081 leader_election.cc:223] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [CANDIDATE]: Term 2 election: Requesting vote from peer 40e545c6b6c540f5b91a210d27d04982 ================== WARNING: ThreadSanitizer: data race (pid=31000) Atomic write of size 8 at 0x7d900004f968 by thread T123: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900004f969 by thread T124: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== I0628 23:55:32.784312 31061 tablet_service.cc:653] Processing DeleteTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:48479 ================== WARNING: ThreadSanitizer: data race (pid=31000) Write of size 8 at 0x7d900004f9e0 by thread T123: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) I0628 23:55:32.784732 31061 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:32.785341 31061 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:32.785545 31061 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:32.785760 31061 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:32.786041 31061 maintenance_manager.cc:180] Unregistered op FlushMRSOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:32.786232 31061 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:32.786412 31061 maintenance_manager.cc:180] Unregistered op LogGCOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:32.786604 31061 raft_consensus.cc:1495] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 2 FOLLOWER]: Raft consensus shutting down. I0628 23:55:32.787046 31061 raft_consensus.cc:1511] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 2 FOLLOWER]: Raft consensus is shut down! I0628 23:55:32.787761 31061 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:55:32.787886 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 7), ts=6009479711107317760, rows=[INSERT (int32 key=1182085342, int32 int_val=937124597, string string_val=hello world), INSERT (int32 key=608034681, int32 int_val=1511691141, string string_val=hello world), INSERT (int32 key=113979130, int32 int_val=91824786, string string_val=hello world), ...]]] I0628 23:55:32.789146 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d4400058200 [op_id=(term: 1 index: 6), ts=6009479710942810112, rows=[INSERT (int32 key=28233841, int32 int_val=2079763347, string string_val=hello world), INSERT (int32 key=2138734457, int32 int_val=1128735313, string string_val=hello world), INSERT (int32 key=1931351640, int32 int_val=1011689075, string string_val=hello world), ...]]] I0628 23:55:32.789902 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d440004fc40 [op_id=(term: 1 index: 5), ts=6009479710573473792, rows=[INSERT (int32 key=3126102, int32 int_val=1000788786, string string_val=hello world), INSERT (int32 key=1165202998, int32 int_val=663410393, string string_val=hello world), INSERT (int32 key=203379927, int32 int_val=1559950712, string string_val=hello world), ...]]] #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900004f9e0 by thread T124: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=31411, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=31414, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=31409, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== I0628 23:55:32.800441 31061 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:55:32.800568 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 7), ts=6009479711107317760, rows=[INSERT (int32 key=1182085342, int32 int_val=937124597, string string_val=hello world), INSERT (int32 key=608034681, int32 int_val=1511691141, string string_val=hello world), INSERT (int32 key=113979130, int32 int_val=91824786, string string_val=hello world), ...]]] I0628 23:55:32.801316 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d440004fc40 [op_id=(term: 1 index: 5), ts=6009479710573473792, rows=[INSERT (int32 key=3126102, int32 int_val=1000788786, string string_val=hello world), INSERT (int32 key=1165202998, int32 int_val=663410393, string string_val=hello world), INSERT (int32 key=203379927, int32 int_val=1559950712, string string_val=hello world), ...]]] I0628 23:55:32.804256 31061 transaction_tracker.cc:230] Dumping currently running transactions: I0628 23:55:32.804396 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 7), ts=6009479711107317760, rows=[INSERT (int32 key=1182085342, int32 int_val=937124597, string string_val=hello world), INSERT (int32 key=608034681, int32 int_val=1511691141, string string_val=hello world), INSERT (int32 key=113979130, int32 int_val=91824786, string string_val=hello world), ...]]] I0628 23:55:32.805117 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d440004fc40 [op_id=(term: 1 index: 5), ts=6009479710573473792, rows=[INSERT (int32 key=3126102, int32 int_val=1000788786, string string_val=hello world), INSERT (int32 key=1165202998, int32 int_val=663410393, string string_val=hello world), INSERT (int32 key=203379927, int32 int_val=1559950712, string string_val=hello world), ...]]] I0628 23:55:32.807015 31061 transaction_tracker.cc:230] Dumping currently running transactions: W0628 23:55:32.806984 31018 leader_election.cc:275] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f0f910d23820467fa1cca6dff8ad5c0c: Network error: Client connection negotiation failed: client connection to 127.106.77.2:35961: connect: Connection refused (error 111) W0628 23:55:32.808260 31016 leader_election.cc:275] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 40e545c6b6c540f5b91a210d27d04982: Network error: Client connection negotiation failed: client connection to 127.106.77.1:38052: connect: Connection refused (error 111) I0628 23:55:32.808491 31016 leader_election.cc:248] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. W0628 23:55:32.808802 31016 raft_consensus.cc:1801] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Unable to run election callback: Service unavailable: The pool has been shut down. I0628 23:55:32.807207 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d44000580c0 [op_id=(term: 1 index: 7), ts=6009479711107317760, rows=[]]] I0628 23:55:32.809083 31061 transaction_tracker.cc:232] RD-P WriteTransaction [type=REPLICA, start_time=2016-06-28 23:55:32, state=WriteTransactionState 0x7d440004fc40 [op_id=(term: 1 index: 5), ts=6009479710573473792, rows=[]]] I0628 23:55:32.816051 31061 ts_tablet_manager.cc:955] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:55:32.820443 31061 ts_tablet_manager.cc:965] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Tablet deleted. Last logged OpId: 1.7 I0628 23:55:32.821395 31061 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Deleted tablet blocks from disk I0628 23:55:32.828723 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 31000 I0628 23:55:32.849459 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:38052 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_port=43953 --webserver_interface=127.106.77.1 --tserver_master_addrs=127.0.0.1:43578 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --enable_leader_failure_detection=false --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:32.967162 31426 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:32.967599 31426 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:32.981740 31426 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:32.984557 31426 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 152351 I0628 23:55:32.996428 31426 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 uuid: "40e545c6b6c540f5b91a210d27d04982" format_stamp: "Formatted at 2016-06-28 23:55:31 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:33.031936 31426 ts_tablet_manager.cc:604] Loading metadata for tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:33.037463 31426 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:33.039299 31436 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrapping tablet I0628 23:55:33.040664 31426 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:33.042989 31436 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrap starting. I0628 23:55:33.049108 31436 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent opening tablet: real 0.002s user 0.000s sys 0.000s I0628 23:55:33.049984 31436 tablet_bootstrap.cc:609] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 I0628 23:55:33.050197 31436 tablet_bootstrap.cc:617] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78 to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery in preparation for log replay W0628 23:55:33.052464 31436 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0628 23:55:33.052781 31436 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:33.097604 31436 log_util.cc:569] Scanning /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 for valid entry headers following offset 14522... I0628 23:55:33.152915 31426 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:38052 I0628 23:55:33.154482 31426 webserver.cc:122] Starting webserver on 127.106.77.1:43953 I0628 23:55:33.154572 31426 webserver.cc:133] Document root disabled I0628 23:55:33.155993 31426 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:43953/ I0628 23:55:33.157425 31426 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:55:33.159193 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 31426 I0628 23:55:33.160166 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:35961 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_port=43897 --webserver_interface=127.106.77.2 --tserver_master_addrs=127.0.0.1:43578 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --enable_leader_failure_detection=false --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:33.168259 31426 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:33.175132 31541 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43578 I0628 23:55:33.175493 31541 heartbeater.cc:319] Registering TS with master... I0628 23:55:33.175750 31541 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:33.177609 30972 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" instance_seqno: 1467158133131556 } with Master I0628 23:55:33.239598 31436 log_util.cc:606] Found no log entry headers I0628 23:55:33.240463 31436 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 14510 for log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Prior entries: [off=11793 COMMIT (1.6)] [off=13848 REPLICATE (1.7)] [off=14179 COMMIT (1.7)] [off=14510 COMMIT (1.5)] I0628 23:55:33.240710 31436 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 (valid entries through byte offset 14510) I0628 23:55:33.241592 31436 tablet.cc:854] T de15e82351f348c0b54b3c21a2cc9c78 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0628 23:55:33.244513 31436 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:33.294509 31545 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:33.294970 31545 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:33.309355 31545 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:33.312391 31545 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 152851 I0628 23:55:33.320026 31436 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:55:33.320336 31436 tablet_bootstrap.cc:1030] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: ReplayState: Previous OpId: 1.7, Committed OpId: 1.7, Pending Replicates: 0, Pending Commits: 0 I0628 23:55:33.325165 31436 tablet_bootstrap.cc:646] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery I0628 23:55:33.325341 31436 tablet_bootstrap.cc:649] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158133325272 I0628 23:55:33.325523 31436 tablet_bootstrap.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158133325272 I0628 23:55:33.326210 31436 tablet_bootstrap.cc:662] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158133325272 I0628 23:55:33.326453 31436 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrap complete. I0628 23:55:33.325299 31545 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 uuid: "f0f910d23820467fa1cca6dff8ad5c0c" format_stamp: "Formatted at 2016-06-28 23:55:31 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:33.332226 31436 ts_tablet_manager.cc:633] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent bootstrapping tablet: real 0.292s user 0.236s sys 0.052s I0628 23:55:33.336967 31436 raft_consensus.cc:269] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:33.338204 31436 raft_consensus.cc:499] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e545c6b6c540f5b91a210d27d04982, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 7 Committed: term: 1 index: 7} I0628 23:55:33.338645 31436 consensus_queue.cc:162] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.7, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:33.351346 31436 ts_tablet_manager.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent starting tablet: real 0.019s user 0.008s sys 0.008s I0628 23:55:33.362293 31545 ts_tablet_manager.cc:604] Loading metadata for tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:33.367904 31545 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:33.369614 31561 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrapping tablet I0628 23:55:33.372093 31545 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:33.374342 31561 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrap starting. I0628 23:55:33.381569 31561 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent opening tablet: real 0.002s user 0.000s sys 0.000s I0628 23:55:33.382515 31561 tablet_bootstrap.cc:609] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 I0628 23:55:33.382688 31561 tablet_bootstrap.cc:617] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78 to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery in preparation for log replay W0628 23:55:33.385030 31561 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0628 23:55:33.385401 31561 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:33.429183 31561 log_util.cc:569] Scanning /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 for valid entry headers following offset 14486... I0628 23:55:33.483608 31545 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:35961 I0628 23:55:33.485203 31545 webserver.cc:122] Starting webserver on 127.106.77.2:43897 I0628 23:55:33.485326 31545 webserver.cc:133] Document root disabled I0628 23:55:33.487082 31545 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:43897/ I0628 23:55:33.488627 31545 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:55:33.491920 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 31545 I0628 23:55:33.510551 31666 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43578 I0628 23:55:33.510895 31545 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:33.511024 31666 heartbeater.cc:319] Registering TS with master... I0628 23:55:33.511330 31666 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:33.514160 30972 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" instance_seqno: 1467158133463085 } with Master I0628 23:55:33.571398 31561 log_util.cc:606] Found no log entry headers I0628 23:55:33.572147 31561 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 14474 for log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Prior entries: [off=13481 REPLICATE (1.7)] [off=13812 COMMIT (1.5)] [off=14143 COMMIT (1.6)] [off=14474 COMMIT (1.7)] I0628 23:55:33.572438 31561 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 (valid entries through byte offset 14474) I0628 23:55:33.573340 31561 tablet.cc:854] T de15e82351f348c0b54b3c21a2cc9c78 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0628 23:55:33.576691 31561 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:33.649466 31561 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:55:33.649809 31561 tablet_bootstrap.cc:1030] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: ReplayState: Previous OpId: 1.7, Committed OpId: 1.7, Pending Replicates: 0, Pending Commits: 0 I0628 23:55:33.653581 31561 tablet_bootstrap.cc:646] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery I0628 23:55:33.653782 31561 tablet_bootstrap.cc:649] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158133653709 I0628 23:55:33.654053 31561 tablet_bootstrap.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158133653709 I0628 23:55:33.654788 31561 tablet_bootstrap.cc:662] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158133653709 I0628 23:55:33.655066 31561 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrap complete. I0628 23:55:33.660544 31561 ts_tablet_manager.cc:633] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent bootstrapping tablet: real 0.291s user 0.240s sys 0.040s I0628 23:55:33.665465 31561 raft_consensus.cc:269] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:33.666595 31561 raft_consensus.cc:499] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f910d23820467fa1cca6dff8ad5c0c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 7 Committed: term: 1 index: 7} I0628 23:55:33.667004 31561 consensus_queue.cc:162] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.7, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:33.669997 31561 ts_tablet_manager.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s I0628 23:55:33.674470 31498 raft_consensus.cc:370] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:33.674685 31498 raft_consensus.cc:2019] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 1 FOLLOWER]: Advancing to term 2 I0628 23:55:33.676775 31498 raft_consensus.cc:386] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:33.681139 31498 leader_election.cc:223] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 2 election: Requesting vote from peer f0f910d23820467fa1cca6dff8ad5c0c I0628 23:55:33.681668 31498 leader_election.cc:223] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 2 election: Requesting vote from peer 13048edf8677461fad00fc37acf1aeb7 W0628 23:55:33.686089 31433 leader_election.cc:275] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 13048edf8677461fad00fc37acf1aeb7: Network error: Client connection negotiation failed: client connection to 127.106.77.0:57634: connect: Connection refused (error 111) I0628 23:55:33.688338 31623 raft_consensus.cc:2019] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 1 FOLLOWER]: Advancing to term 2 I0628 23:55:33.692193 31623 raft_consensus.cc:1677] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e545c6b6c540f5b91a210d27d04982 in term 2. I0628 23:55:33.693238 31434 leader_election.cc:347] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 2 election: Vote granted by peer f0f910d23820467fa1cca6dff8ad5c0c I0628 23:55:33.693444 31434 leader_election.cc:248] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0628 23:55:33.693979 31555 raft_consensus.cc:1859] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 FOLLOWER]: Leader election won for term 2 I0628 23:55:33.695657 31555 raft_consensus.cc:464] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 LEADER]: Becoming Leader. State: Replica: 40e545c6b6c540f5b91a210d27d04982, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 7 Committed: term: 1 index: 7} I0628 23:55:33.696202 31555 consensus_queue.cc:145] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 1.7, Committed index: 1.7, Last appended: 1.7, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:33.703117 27213 delete_table-test.cc:652] Bringing TS 13048edf8677461fad00fc37acf1aeb7 back up... I0628 23:55:33.703804 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:57634 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=53688 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:43578 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0 --enable_leader_failure_detection=false --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 W0628 23:55:33.704722 31678 raft_consensus_state.cc:541] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 1 index: 7, New majority replicated is: term: 1 index: 7, Current term is: 2 I0628 23:55:33.704356 30972 catalog_manager.cc:1673] Tablet: de15e82351f348c0b54b3c21a2cc9c78 reported consensus state change. New consensus state: current_term: 2 leader_uuid: "40e545c6b6c540f5b91a210d27d04982" config { opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } } I0628 23:55:33.826678 31679 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:33.827262 31679 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:33.841138 31679 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:33.844050 31679 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 152851 I0628 23:55:33.856858 31679 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0 uuid: "13048edf8677461fad00fc37acf1aeb7" format_stamp: "Formatted at 2016-06-28 23:55:30 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:33.891875 31679 ts_tablet_manager.cc:604] Loading metadata for tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:33.897028 31679 ts_tablet_manager.cc:936] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_TOMBSTONED I0628 23:55:33.903515 31679 ts_tablet_manager.cc:955] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:55:33.909292 31679 ts_tablet_manager.cc:965] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Tablet deleted. Last logged OpId: 1.7 I0628 23:55:33.909828 31679 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:33.910972 31679 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:34.002640 31679 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:57634 I0628 23:55:34.004392 31679 webserver.cc:122] Starting webserver on 127.106.77.0:53688 I0628 23:55:34.004489 31679 webserver.cc:133] Document root disabled I0628 23:55:34.006007 31679 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:53688/ I0628 23:55:34.007261 31679 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:34.010077 31679 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:34.013999 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 31679 I0628 23:55:34.019914 31793 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43578 I0628 23:55:34.020303 31793 heartbeater.cc:319] Registering TS with master... I0628 23:55:34.020519 31793 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:34.022084 30972 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" instance_seqno: 1467158133983915 } with Master I0628 23:55:34.202857 31623 raft_consensus.cc:869] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 2 FOLLOWER]: Refusing update from remote peer 40e545c6b6c540f5b91a210d27d04982: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch) I0628 23:55:34.205312 31555 consensus_queue.cc:578] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [LEADER]: Connected to new peer: Peer: f0f910d23820467fa1cca6dff8ad5c0c, Is new: false, Last received: 1.7, Next index: 8, Last known committed idx: 7, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:55:34.211835 31555 consensus_peers.cc:296] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 -> Peer 13048edf8677461fad00fc37acf1aeb7 (127.106.77.0:57634): Sending request to remotely bootstrap I0628 23:55:34.213870 31750 ts_tablet_manager.cc:412] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Initiating remote bootstrap from Peer 40e545c6b6c540f5b91a210d27d04982 (127.106.77.1:38052) I0628 23:55:34.216835 31750 remote_bootstrap_client.cc:164] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Remote bootstrap client: Beginning remote bootstrap session from remote peer at address 127.106.77.1:38052 I0628 23:55:34.219015 31799 raft_consensus_state.cc:534] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 1 index: 7 New committed index is: term: 2 index: 8 I0628 23:55:34.222935 31518 remote_bootstrap_service.cc:123] Beginning new remote bootstrap session on tablet de15e82351f348c0b54b3c21a2cc9c78 from peer 13048edf8677461fad00fc37acf1aeb7 at {real_user=slave, eff_user=} at 127.106.77.0:50865: session id = 13048edf8677461fad00fc37acf1aeb7-de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:34.226896 31518 remote_bootstrap_session.cc:142] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Remote bootstrap: opened 0 blocks and 1 log segments I0628 23:55:34.233683 31750 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:34.234153 31750 remote_bootstrap_client.cc:387] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Remote bootstrap client: Starting download of 0 data blocks... I0628 23:55:34.234477 31750 remote_bootstrap_client.cc:352] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Remote bootstrap client: Starting download of 1 WAL segments... I0628 23:55:34.234675 31750 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: RemoteBootstrap: Downloading WAL segment with seq. number 1 (1/1) I0628 23:55:34.241806 31750 remote_bootstrap_client.cc:273] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Remote bootstrap client: Remote bootstrap complete. Replacing tablet superblock. I0628 23:55:34.241942 31750 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: RemoteBootstrap: Replacing tablet superblock I0628 23:55:34.245954 31800 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Bootstrapping tablet I0628 23:55:34.246700 31518 remote_bootstrap_service.cc:242] Request end of remote bootstrap session 13048edf8677461fad00fc37acf1aeb7-de15e82351f348c0b54b3c21a2cc9c78 received from {real_user=slave, eff_user=} at 127.106.77.0:50865 I0628 23:55:34.246906 31518 remote_bootstrap_service.cc:323] Ending remote bootstrap session 13048edf8677461fad00fc37acf1aeb7-de15e82351f348c0b54b3c21a2cc9c78 on tablet de15e82351f348c0b54b3c21a2cc9c78 with peer 13048edf8677461fad00fc37acf1aeb7 I0628 23:55:34.248472 31800 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Bootstrap starting. I0628 23:55:34.249719 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 31426 I0628 23:55:34.253082 31800 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:34.253738 31800 tablet_bootstrap.cc:609] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 I0628 23:55:34.253875 31800 tablet_bootstrap.cc:617] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78 to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery in preparation for log replay W0628 23:55:34.255162 31800 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0628 23:55:34.255379 31800 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:34.259620 31800 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 (valid entries through byte offset 14574) I0628 23:55:34.260485 31800 tablet.cc:854] T de15e82351f348c0b54b3c21a2cc9c78 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0628 23:55:34.263173 31800 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:34.269278 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 31545 I0628 23:55:34.290796 31730 tablet_service.cc:653] Processing DeleteTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:48485 I0628 23:55:34.302963 31730 tablet_service.cc:653] Processing DeleteTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:48485 I0628 23:55:34.314908 31730 tablet_service.cc:653] Processing DeleteTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:48485 I0628 23:55:34.326809 31730 tablet_service.cc:653] Processing DeleteTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:48485 I0628 23:55:34.338699 31730 tablet_service.cc:653] Processing DeleteTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:48485 I0628 23:55:34.341090 31800 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:55:34.341318 31800 tablet_bootstrap.cc:1030] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: ReplayState: Previous OpId: 2.8, Committed OpId: 2.8, Pending Replicates: 0, Pending Commits: 0 I0628 23:55:34.343235 31800 tablet_bootstrap.cc:646] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery I0628 23:55:34.343428 31800 tablet_bootstrap.cc:649] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158134343355 I0628 23:55:34.343575 31800 tablet_bootstrap.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158134343355 I0628 23:55:34.344355 31800 tablet_bootstrap.cc:662] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158134343355 I0628 23:55:34.344578 31800 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Bootstrap complete. I0628 23:55:34.344933 31800 ts_tablet_manager.cc:633] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Time spent bootstrapping tablet: real 0.099s user 0.088s sys 0.008s I0628 23:55:34.350694 31730 tablet_service.cc:653] Processing DeleteTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:48485 I0628 23:55:34.349884 31800 raft_consensus.cc:269] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:34.351663 31800 raft_consensus.cc:499] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13048edf8677461fad00fc37acf1aeb7, State: 1, Role: FOLLOWER Watermarks: {Received: term: 2 index: 8 Committed: term: 2 index: 8} I0628 23:55:34.352166 31800 consensus_queue.cc:162] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 2.8, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:34.355226 31800 ts_tablet_manager.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s I0628 23:55:34.362849 31730 tablet_service.cc:653] Processing DeleteTablet for tablet de15e82351f348c0b54b3c21a2cc9c78 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:48485 I0628 23:55:34.363092 31730 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:34.363507 31730 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:34.363687 31730 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:34.363837 31730 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:34.364176 31730 maintenance_manager.cc:180] Unregistered op FlushMRSOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:34.364310 31730 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:34.364429 31730 maintenance_manager.cc:180] Unregistered op LogGCOp(de15e82351f348c0b54b3c21a2cc9c78) I0628 23:55:34.364558 31730 raft_consensus.cc:1495] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 2 FOLLOWER]: Raft consensus shutting down. I0628 23:55:34.364872 31730 raft_consensus.cc:1511] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7 [term 2 FOLLOWER]: Raft consensus is shut down! I0628 23:55:34.369096 31730 ts_tablet_manager.cc:955] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:55:34.373227 31730 ts_tablet_manager.cc:965] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Tablet deleted. Last logged OpId: 2.8 I0628 23:55:34.373823 31730 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Deleted tablet blocks from disk I0628 23:55:34.378536 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:38052 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_port=43953 --webserver_interface=127.106.77.1 --tserver_master_addrs=127.0.0.1:43578 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 --enable_leader_failure_detection=false --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:34.493659 31805 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:34.494107 31805 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:34.507016 31805 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:34.510041 31805 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 153351 I0628 23:55:34.521657 31805 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1 uuid: "40e545c6b6c540f5b91a210d27d04982" format_stamp: "Formatted at 2016-06-28 23:55:31 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:34.556702 31805 ts_tablet_manager.cc:604] Loading metadata for tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:34.562851 31805 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:34.564452 31815 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrapping tablet I0628 23:55:34.565582 31805 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:34.568354 31815 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrap starting. I0628 23:55:34.573513 31815 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent opening tablet: real 0.002s user 0.004s sys 0.000s I0628 23:55:34.574434 31815 tablet_bootstrap.cc:609] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 I0628 23:55:34.574566 31815 tablet_bootstrap.cc:617] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78 to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery in preparation for log replay W0628 23:55:34.576503 31815 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0628 23:55:34.576808 31815 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:34.628161 31815 log_util.cc:569] Scanning /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 for valid entry headers following offset 14586... I0628 23:55:34.681557 31805 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:38052 I0628 23:55:34.683064 31805 webserver.cc:122] Starting webserver on 127.106.77.1:43953 I0628 23:55:34.683164 31805 webserver.cc:133] Document root disabled I0628 23:55:34.684718 31805 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:43953/ I0628 23:55:34.686185 31805 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:55:34.687187 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 31805 I0628 23:55:34.688242 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:35961 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_port=43897 --webserver_interface=127.106.77.2 --tserver_master_addrs=127.0.0.1:43578 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 --enable_leader_failure_detection=false --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:34.698170 31805 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:34.705340 31920 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43578 I0628 23:55:34.705767 31920 heartbeater.cc:319] Registering TS with master... I0628 23:55:34.706058 31920 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:34.708400 30972 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" instance_seqno: 1467158134657962 } with Master I0628 23:55:34.749771 31815 log_util.cc:606] Found no log entry headers I0628 23:55:34.750715 31815 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 14574 for log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Prior entries: [off=14179 COMMIT (1.6)] [off=14510 COMMIT (1.7)] [off=14548 REPLICATE (2.8)] [off=14574 COMMIT (2.8)] I0628 23:55:34.751044 31815 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 (valid entries through byte offset 14574) I0628 23:55:34.751782 31815 tablet.cc:854] T de15e82351f348c0b54b3c21a2cc9c78 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0628 23:55:34.754637 31815 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:34.825661 31815 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:55:34.825938 31815 tablet_bootstrap.cc:1030] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: ReplayState: Previous OpId: 2.8, Committed OpId: 2.8, Pending Replicates: 0, Pending Commits: 0 I0628 23:55:34.829318 31815 tablet_bootstrap.cc:646] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery I0628 23:55:34.829493 31815 tablet_bootstrap.cc:649] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158134829417 I0628 23:55:34.829663 31815 tablet_bootstrap.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158134829417 I0628 23:55:34.829416 31924 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:34.830118 31924 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:34.830471 31815 tablet_bootstrap.cc:662] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-1/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158134829417 I0628 23:55:34.830705 31815 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Bootstrap complete. I0628 23:55:34.836336 31815 ts_tablet_manager.cc:633] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent bootstrapping tablet: real 0.272s user 0.216s sys 0.052s I0628 23:55:34.840880 31815 raft_consensus.cc:269] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:34.842069 31815 raft_consensus.cc:499] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e545c6b6c540f5b91a210d27d04982, State: 1, Role: FOLLOWER Watermarks: {Received: term: 2 index: 8 Committed: term: 2 index: 8} I0628 23:55:34.842690 31815 consensus_queue.cc:162] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 2.8, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:34.846698 31815 ts_tablet_manager.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s I0628 23:55:34.857041 31924 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:34.860973 31924 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 153351 I0628 23:55:34.873754 31924 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2 uuid: "f0f910d23820467fa1cca6dff8ad5c0c" format_stamp: "Formatted at 2016-06-28 23:55:31 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:34.928997 31924 ts_tablet_manager.cc:604] Loading metadata for tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:34.934669 31924 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:34.944882 31937 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrapping tablet I0628 23:55:34.948019 31924 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:34.948611 31937 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrap starting. I0628 23:55:34.954704 31937 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent opening tablet: real 0.002s user 0.000s sys 0.000s I0628 23:55:34.955724 31937 tablet_bootstrap.cc:609] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 I0628 23:55:34.955899 31937 tablet_bootstrap.cc:617] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78 to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery in preparation for log replay W0628 23:55:34.957947 31937 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0628 23:55:34.958219 31937 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:35.009251 31937 log_util.cc:569] Scanning /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 for valid entry headers following offset 14586... I0628 23:55:35.056738 31924 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:35961 I0628 23:55:35.059137 31924 webserver.cc:122] Starting webserver on 127.106.77.2:43897 I0628 23:55:35.059270 31924 webserver.cc:133] Document root disabled I0628 23:55:35.061254 31924 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:43897/ I0628 23:55:35.063241 31924 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:55:35.066220 31924 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:35.071490 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 31924 I0628 23:55:35.077406 32042 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43578 I0628 23:55:35.077828 32042 heartbeater.cc:319] Registering TS with master... I0628 23:55:35.078096 32042 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:35.079886 30972 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" instance_seqno: 1467158135034964 } with Master I0628 23:55:35.149137 31937 log_util.cc:606] Found no log entry headers I0628 23:55:35.149852 31937 log_util.cc:215] Ignoring log segment corruption in /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via remote bootstrap. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 14574 for log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Prior entries: [off=14179 COMMIT (1.6)] [off=14510 COMMIT (1.7)] [off=14548 REPLICATE (2.8)] [off=14574 COMMIT (2.8)] I0628 23:55:35.150123 31937 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 (valid entries through byte offset 14574) I0628 23:55:35.150957 31937 tablet.cc:854] T de15e82351f348c0b54b3c21a2cc9c78 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0628 23:55:35.154076 31937 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:35.228253 31937 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0628 23:55:35.228548 31937 tablet_bootstrap.cc:1030] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: ReplayState: Previous OpId: 2.8, Committed OpId: 2.8, Pending Replicates: 0, Pending Commits: 0 I0628 23:55:35.232290 31937 tablet_bootstrap.cc:646] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Preparing to delete log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery I0628 23:55:35.232442 31937 tablet_bootstrap.cc:649] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Renaming log recovery dir from /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158135232377 I0628 23:55:35.232584 31937 tablet_bootstrap.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Deleting all files from renamed log recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158135232377 I0628 23:55:35.233332 31937 tablet_bootstrap.cc:662] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Completed deletion of old log recovery files and directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-2/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery-1467158135232377 I0628 23:55:35.233538 31937 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Bootstrap complete. I0628 23:55:35.238689 31937 ts_tablet_manager.cc:633] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent bootstrapping tablet: real 0.293s user 0.244s sys 0.044s I0628 23:55:35.243036 31937 raft_consensus.cc:269] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:35.244065 31937 raft_consensus.cc:499] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f910d23820467fa1cca6dff8ad5c0c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 2 index: 8 Committed: term: 2 index: 8} I0628 23:55:35.244563 31937 consensus_queue.cc:162] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 2.8, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:35.247511 31937 ts_tablet_manager.cc:659] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s I0628 23:55:35.252295 31877 raft_consensus.cc:370] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:35.252465 31877 raft_consensus.cc:2019] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 2 FOLLOWER]: Advancing to term 3 I0628 23:55:35.255094 31877 raft_consensus.cc:386] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:35.258443 31877 leader_election.cc:223] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 3 election: Requesting vote from peer f0f910d23820467fa1cca6dff8ad5c0c I0628 23:55:35.258858 31877 leader_election.cc:223] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 3 election: Requesting vote from peer 13048edf8677461fad00fc37acf1aeb7 I0628 23:55:35.269175 31999 raft_consensus.cc:2019] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 2 FOLLOWER]: Advancing to term 3 W0628 23:55:35.271361 31812 leader_election.cc:281] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer 13048edf8677461fad00fc37acf1aeb7: Not found: Tablet not found I0628 23:55:35.273669 31999 raft_consensus.cc:1677] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e545c6b6c540f5b91a210d27d04982 in term 3. I0628 23:55:35.274670 31813 leader_election.cc:347] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 3 election: Vote granted by peer f0f910d23820467fa1cca6dff8ad5c0c I0628 23:55:35.274863 31813 leader_election.cc:248] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0628 23:55:35.275396 31928 raft_consensus.cc:1859] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 3 FOLLOWER]: Leader election won for term 3 I0628 23:55:35.276722 31928 raft_consensus.cc:464] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 3 LEADER]: Becoming Leader. State: Replica: 40e545c6b6c540f5b91a210d27d04982, State: 1, Role: LEADER Watermarks: {Received: term: 2 index: 8 Committed: term: 2 index: 8} I0628 23:55:35.277539 31928 consensus_queue.cc:145] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 2.8, Committed index: 2.8, Last appended: 2.8, Current term: 3, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } I0628 23:55:35.286269 30972 catalog_manager.cc:1673] Tablet: de15e82351f348c0b54b3c21a2cc9c78 reported consensus state change. New consensus state: current_term: 3 leader_uuid: "40e545c6b6c540f5b91a210d27d04982" config { opid_index: -1 peers { permanent_uuid: "f0f910d23820467fa1cca6dff8ad5c0c" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 35961 } } peers { permanent_uuid: "13048edf8677461fad00fc37acf1aeb7" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 57634 } } peers { permanent_uuid: "40e545c6b6c540f5b91a210d27d04982" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 38052 } } } W0628 23:55:35.287145 32054 raft_consensus_state.cc:541] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 3 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 8, New majority replicated is: term: 2 index: 8, Current term is: 3 I0628 23:55:35.784482 31999 raft_consensus.cc:869] T de15e82351f348c0b54b3c21a2cc9c78 P f0f910d23820467fa1cca6dff8ad5c0c [term 3 FOLLOWER]: Refusing update from remote peer 40e545c6b6c540f5b91a210d27d04982: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0628 23:55:35.785826 31928 consensus_peers.cc:296] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 -> Peer 13048edf8677461fad00fc37acf1aeb7 (127.106.77.0:57634): Sending request to remotely bootstrap I0628 23:55:35.786916 31928 consensus_queue.cc:578] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [LEADER]: Connected to new peer: Peer: f0f910d23820467fa1cca6dff8ad5c0c, Is new: false, Last received: 2.8, Next index: 9, Last known committed idx: 8, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:55:35.787232 31750 ts_tablet_manager.cc:412] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Initiating remote bootstrap from Peer 40e545c6b6c540f5b91a210d27d04982 (127.106.77.1:38052) I0628 23:55:35.789265 31750 remote_bootstrap_client.cc:164] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Remote bootstrap client: Beginning remote bootstrap session from remote peer at address 127.106.77.1:38052 I0628 23:55:35.809031 31897 remote_bootstrap_service.cc:123] Beginning new remote bootstrap session on tablet de15e82351f348c0b54b3c21a2cc9c78 from peer 13048edf8677461fad00fc37acf1aeb7 at {real_user=slave, eff_user=} at 127.106.77.0:52895: session id = 13048edf8677461fad00fc37acf1aeb7-de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:35.812487 31897 remote_bootstrap_session.cc:142] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982: Remote bootstrap: opened 0 blocks and 1 log segments I0628 23:55:35.819149 31750 ts_tablet_manager.cc:760] Registered tablet de15e82351f348c0b54b3c21a2cc9c78 I0628 23:55:35.819371 31750 remote_bootstrap_client.cc:387] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Remote bootstrap client: Starting download of 0 data blocks... I0628 23:55:35.819581 31750 remote_bootstrap_client.cc:352] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Remote bootstrap client: Starting download of 1 WAL segments... I0628 23:55:35.819710 31750 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: RemoteBootstrap: Downloading WAL segment with seq. number 1 (1/1) I0628 23:55:35.824849 31750 remote_bootstrap_client.cc:273] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Remote bootstrap client: Remote bootstrap complete. Replacing tablet superblock. I0628 23:55:35.825042 31750 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: RemoteBootstrap: Replacing tablet superblock I0628 23:55:35.831334 32061 raft_consensus_state.cc:534] T de15e82351f348c0b54b3c21a2cc9c78 P 40e545c6b6c540f5b91a210d27d04982 [term 3 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 2 index: 8 New committed index is: term: 3 index: 9 I0628 23:55:35.844022 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 30956 I0628 23:55:35.844694 32062 ts_tablet_manager.cc:628] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Bootstrapping tablet I0628 23:55:35.845471 31897 remote_bootstrap_service.cc:242] Request end of remote bootstrap session 13048edf8677461fad00fc37acf1aeb7-de15e82351f348c0b54b3c21a2cc9c78 received from {real_user=slave, eff_user=} at 127.106.77.0:52895 I0628 23:55:35.845625 31897 remote_bootstrap_service.cc:323] Ending remote bootstrap session 13048edf8677461fad00fc37acf1aeb7-de15e82351f348c0b54b3c21a2cc9c78 on tablet de15e82351f348c0b54b3c21a2cc9c78 with peer 13048edf8677461fad00fc37acf1aeb7 I0628 23:55:35.847875 32062 tablet_bootstrap.cc:393] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Bootstrap starting. I0628 23:55:35.850452 32062 tablet_bootstrap.cc:553] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:35.851049 32062 tablet_bootstrap.cc:609] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Will attempt to recover log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78/wal-000000001 to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 I0628 23:55:35.851222 32062 tablet_bootstrap.cc:617] T de15e82351f348c0b54b3c21a2cc9c78 P 13048edf8677461fad00fc37acf1aeb7: Moving log directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78 to recovery directory /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery in preparation for log replay W0628 23:55:35.852013 32062 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0628 23:55:35.852205 32062 log_reader.cc:152] Log segment /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0628 23:55:35.858085 32062 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestMergeConsensusMetadata.1467158082783852-27213/minicluster-data/ts-0/wals/de15e82351f348c0b54b3c21a2cc9c78.recovery/wal-000000001 (valid entries through byte offset 14612) I0628 23:55:35.858618 32062 tablet.cc:854] T de15e82351f348c0b54b3c21a2cc9c78 Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] W0628 23:55:35.870446 32042 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:43578: Network error: Failed to send heartbeat: Recv() got EOF from remote (error 108) I0628 23:55:35.871484 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 31679 I0628 23:55:35.887882 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 31805 I0628 23:55:35.904860 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 31924 [ OK ] DeleteTableTest.TestMergeConsensusMetadata (5339 ms) [ RUN ] DeleteTableTest.TestDeleteFollowerWithReplicatingTransaction I0628 23:55:35.928200 27213 delete_table-test.cc:692] Skipping test in fast-test mode. [ OK ] DeleteTableTest.TestDeleteFollowerWithReplicatingTransaction (1 ms) [ RUN ] DeleteTableTest.TestOrphanedBlocksClearedOnDelete I0628 23:55:35.930882 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/master --catalog_manager_wait_for_new_tablets_to_elect_leader=false --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:36.047832 32067 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:36.048334 32067 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:36.061548 32067 master_main.cc:58] Initializing master server... I0628 23:55:36.063809 32067 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 154351 I0628 23:55:36.070866 32067 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:55:36.071100 32067 server_base.cc:165] Creating new FS layout I0628 23:55:36.084481 32067 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/master/instance: uuid: "38fae6210e3b43d19d7e0c21fa9a9984" format_stamp: "Formatted at 2016-06-28 23:55:36 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:36.097512 32067 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/master uuid: "38fae6210e3b43d19d7e0c21fa9a9984" format_stamp: "Formatted at 2016-06-28 23:55:36 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:36.130661 32067 master_main.cc:61] Starting Master server... I0628 23:55:36.164651 32067 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:56838 I0628 23:55:36.166323 32067 webserver.cc:122] Starting webserver on localhost:0 I0628 23:55:36.166435 32067 webserver.cc:133] Document root disabled I0628 23:55:36.167219 32067 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:55:36.168447 32067 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:45660/ I0628 23:55:36.169725 32067 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:55:36.178488 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 32067 I0628 23:55:36.179460 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:56838 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-0 --enable_leader_failure_detection=false --flush_threshold_mb=0 --maintenance_manager_polling_interval_ms=100 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:36.198981 32110 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984: Bootstrap starting. I0628 23:55:36.203044 32110 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:36.203547 32110 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984: No blocks or log segments found. Creating new log. I0628 23:55:36.205210 32110 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:36.209982 32110 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984: No bootstrap required, opened a new log I0628 23:55:36.216523 32110 raft_consensus.cc:269] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" member_type: VOTER } I0628 23:55:36.217566 32110 raft_consensus.cc:301] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:36.217787 32110 raft_consensus.cc:499] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fae6210e3b43d19d7e0c21fa9a9984, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:36.218147 32110 consensus_queue.cc:162] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:36.218451 32110 raft_consensus.cc:314] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:55:36.218588 32110 raft_consensus.cc:370] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:36.218699 32110 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:36.220340 32110 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.619s I0628 23:55:36.220571 32110 raft_consensus.cc:386] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" member_type: VOTER } I0628 23:55:36.223484 32110 leader_election.cc:248] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:36.225267 32115 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.567s I0628 23:55:36.225564 32115 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:36.228034 32115 raft_consensus.cc:464] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 1 LEADER]: Becoming Leader. State: Replica: 38fae6210e3b43d19d7e0c21fa9a9984, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:36.228869 32115 consensus_queue.cc:145] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" member_type: VOTER } I0628 23:55:36.248992 32110 sys_catalog.cc:235] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" config { opid_index: -1 peers { permanent_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" member_type: VOTER } } I0628 23:55:36.249186 32116 sys_catalog.cc:235] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fae6210e3b43d19d7e0c21fa9a9984. Latest consensus state: current_term: 1 leader_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" config { opid_index: -1 peers { permanent_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" member_type: VOTER } } I0628 23:55:36.250151 32116 sys_catalog.cc:238] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [sys.catalog]: This master's current role is: LEADER I0628 23:55:36.250548 32110 sys_catalog.cc:238] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [sys.catalog]: This master's current role is: LEADER I0628 23:55:36.254218 32119 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:55:36.254839 32110 sys_catalog.cc:311] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:55:36.255645 32118 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:55:36.250972 32117 sys_catalog.cc:235] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" config { opid_index: -1 peers { permanent_uuid: "38fae6210e3b43d19d7e0c21fa9a9984" member_type: VOTER } } I0628 23:55:36.256085 32117 sys_catalog.cc:238] T 00000000000000000000000000000000 P 38fae6210e3b43d19d7e0c21fa9a9984 [sys.catalog]: This master's current role is: LEADER I0628 23:55:36.256404 32067 master_main.cc:64] Master server successfully started. I0628 23:55:36.308732 32111 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:36.309248 32111 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:36.326611 32111 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:36.329609 32111 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 154351 I0628 23:55:36.336226 32111 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:55:36.336452 32111 server_base.cc:165] Creating new FS layout I0628 23:55:36.345525 32111 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "cae3c5da9948493a92c33eee962172af" format_stamp: "Formatted at 2016-06-28 23:55:36 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:36.362515 32111 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-0 uuid: "cae3c5da9948493a92c33eee962172af" format_stamp: "Formatted at 2016-06-28 23:55:36 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:36.440871 32111 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:36.772825 32111 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:42671 I0628 23:55:36.774356 32111 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:55:36.774471 32111 webserver.cc:133] Document root disabled I0628 23:55:36.775985 32111 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:53143/ I0628 23:55:36.777330 32111 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:36.780061 32111 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:36.781008 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 32111 I0628 23:55:36.781919 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:56838 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-1 --enable_leader_failure_detection=false --flush_threshold_mb=0 --maintenance_manager_polling_interval_ms=100 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:36.798966 32235 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:56838 I0628 23:55:36.799507 32235 heartbeater.cc:319] Registering TS with master... I0628 23:55:36.799831 32235 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:36.802170 32087 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "cae3c5da9948493a92c33eee962172af" instance_seqno: 1467158136750157 } with Master I0628 23:55:36.910645 32239 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:36.911157 32239 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:36.925123 32239 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:36.928081 32239 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 154351 I0628 23:55:36.934278 32239 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:55:36.934535 32239 server_base.cc:165] Creating new FS layout I0628 23:55:36.943137 32239 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "1edad4aefcff40b38e3ae1170d2d724c" format_stamp: "Formatted at 2016-06-28 23:55:36 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:36.952024 32239 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-1 uuid: "1edad4aefcff40b38e3ae1170d2d724c" format_stamp: "Formatted at 2016-06-28 23:55:36 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:36.988135 32239 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:37.084997 32239 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:48364 I0628 23:55:37.086449 32239 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:55:37.086560 32239 webserver.cc:133] Document root disabled I0628 23:55:37.088048 32239 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:50595/ I0628 23:55:37.089349 32239 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:55:37.091960 32239 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:37.092891 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 32239 I0628 23:55:37.094244 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:56838 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-2 --enable_leader_failure_detection=false --flush_threshold_mb=0 --maintenance_manager_polling_interval_ms=100 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:37.102326 32353 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:56838 I0628 23:55:37.102840 32353 heartbeater.cc:319] Registering TS with master... I0628 23:55:37.103168 32353 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:37.104684 32087 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "1edad4aefcff40b38e3ae1170d2d724c" instance_seqno: 1467158137065508 } with Master I0628 23:55:37.214205 32356 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:37.214968 32356 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:37.229107 32356 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:37.232347 32356 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 154851 I0628 23:55:37.238306 32356 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:55:37.238494 32356 server_base.cc:165] Creating new FS layout I0628 23:55:37.247294 32356 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "a48b3d8066e2410bb967d2167f2dbdb4" format_stamp: "Formatted at 2016-06-28 23:55:37 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:37.255452 32356 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-2 uuid: "a48b3d8066e2410bb967d2167f2dbdb4" format_stamp: "Formatted at 2016-06-28 23:55:37 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:37.291142 32356 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:37.388571 32356 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:34011 I0628 23:55:37.390025 32356 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:55:37.390167 32356 webserver.cc:133] Document root disabled I0628 23:55:37.391641 32356 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:50717/ I0628 23:55:37.393003 32356 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestOrphanedBlocksClearedOnDelete.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:55:37.394410 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 32356 I0628 23:55:37.395576 32356 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:37.406765 32470 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:56838 I0628 23:55:37.407240 32470 heartbeater.cc:319] Registering TS with master... I0628 23:55:37.407531 32470 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:37.408954 32086 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "a48b3d8066e2410bb967d2167f2dbdb4" instance_seqno: 1467158137369197 } with Master I0628 23:55:37.411924 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:55:37.437597 32086 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:44333: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:55:37.486969 32290 tablet_service.cc:612] Processing CreateTablet for tablet 4815feb633554bff9c217d459d84e0a4 (table=test-workload [id=e3a94186bb2a4a7d8dbcbe69f8678bac]), partition=range: [(), ()) I0628 23:55:37.488797 32407 tablet_service.cc:612] Processing CreateTablet for tablet 4815feb633554bff9c217d459d84e0a4 (table=test-workload [id=e3a94186bb2a4a7d8dbcbe69f8678bac]), partition=range: [(), ()) I0628 23:55:37.490721 32172 tablet_service.cc:612] Processing CreateTablet for tablet 4815feb633554bff9c217d459d84e0a4 (table=test-workload [id=e3a94186bb2a4a7d8dbcbe69f8678bac]), partition=range: [(), ()) I0628 23:55:37.504957 32290 ts_tablet_manager.cc:760] Registered tablet 4815feb633554bff9c217d459d84e0a4 I0628 23:55:37.506897 32490 ts_tablet_manager.cc:628] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c: Bootstrapping tablet I0628 23:55:37.511703 32407 ts_tablet_manager.cc:760] Registered tablet 4815feb633554bff9c217d459d84e0a4 I0628 23:55:37.512142 32490 tablet_bootstrap.cc:393] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c: Bootstrap starting. I0628 23:55:37.513387 32172 ts_tablet_manager.cc:760] Registered tablet 4815feb633554bff9c217d459d84e0a4 I0628 23:55:37.517607 32490 tablet_bootstrap.cc:553] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:55:37.518736 32490 tablet_bootstrap.cc:493] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c: No blocks or log segments found. Creating new log. I0628 23:55:37.522200 32490 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:37.522390 32493 ts_tablet_manager.cc:628] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: Bootstrapping tablet I0628 23:55:37.523875 32492 ts_tablet_manager.cc:628] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4: Bootstrapping tablet I0628 23:55:37.525552 32493 tablet_bootstrap.cc:393] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: Bootstrap starting. I0628 23:55:37.527290 32492 tablet_bootstrap.cc:393] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4: Bootstrap starting. I0628 23:55:37.529355 32493 tablet_bootstrap.cc:553] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:37.529883 32493 tablet_bootstrap.cc:493] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: No blocks or log segments found. Creating new log. I0628 23:55:37.532276 32493 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:37.533234 32492 tablet_bootstrap.cc:553] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:55:37.534114 32492 tablet_bootstrap.cc:493] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4: No blocks or log segments found. Creating new log. I0628 23:55:37.536685 32492 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:37.539446 32490 tablet_bootstrap.cc:393] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c: No bootstrap required, opened a new log I0628 23:55:37.539916 32490 ts_tablet_manager.cc:633] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.000s I0628 23:55:37.543274 32492 tablet_bootstrap.cc:393] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4: No bootstrap required, opened a new log I0628 23:55:37.543702 32492 ts_tablet_manager.cc:633] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s I0628 23:55:37.544087 32493 tablet_bootstrap.cc:393] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: No bootstrap required, opened a new log I0628 23:55:37.544567 32493 ts_tablet_manager.cc:633] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s I0628 23:55:37.547391 32490 raft_consensus.cc:269] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "cae3c5da9948493a92c33eee962172af" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 42671 } } peers { permanent_uuid: "1edad4aefcff40b38e3ae1170d2d724c" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 48364 } } peers { permanent_uuid: "a48b3d8066e2410bb967d2167f2dbdb4" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 34011 } } I0628 23:55:37.548352 32492 raft_consensus.cc:269] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "cae3c5da9948493a92c33eee962172af" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 42671 } } peers { permanent_uuid: "1edad4aefcff40b38e3ae1170d2d724c" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 48364 } } peers { permanent_uuid: "a48b3d8066e2410bb967d2167f2dbdb4" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 34011 } } I0628 23:55:37.549235 32490 raft_consensus.cc:499] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edad4aefcff40b38e3ae1170d2d724c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:37.549484 32492 raft_consensus.cc:499] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48b3d8066e2410bb967d2167f2dbdb4, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:37.549837 32492 consensus_queue.cc:162] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:37.549929 32490 consensus_queue.cc:162] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:37.551596 32493 raft_consensus.cc:269] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "cae3c5da9948493a92c33eee962172af" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 42671 } } peers { permanent_uuid: "1edad4aefcff40b38e3ae1170d2d724c" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 48364 } } peers { permanent_uuid: "a48b3d8066e2410bb967d2167f2dbdb4" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 34011 } } I0628 23:55:37.553436 32493 raft_consensus.cc:499] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae3c5da9948493a92c33eee962172af, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:37.554003 32493 consensus_queue.cc:162] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:37.554605 32492 ts_tablet_manager.cc:659] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s I0628 23:55:37.557207 32490 ts_tablet_manager.cc:659] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s I0628 23:55:37.559559 32493 ts_tablet_manager.cc:659] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s I0628 23:55:37.592838 32310 raft_consensus.cc:370] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:37.593055 32310 raft_consensus.cc:2019] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:37.595255 32310 raft_consensus.cc:386] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "cae3c5da9948493a92c33eee962172af" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 42671 } } peers { permanent_uuid: "1edad4aefcff40b38e3ae1170d2d724c" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 48364 } } peers { permanent_uuid: "a48b3d8066e2410bb967d2167f2dbdb4" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 34011 } } I0628 23:55:37.598902 32310 leader_election.cc:223] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [CANDIDATE]: Term 1 election: Requesting vote from peer cae3c5da9948493a92c33eee962172af I0628 23:55:37.599431 32310 leader_election.cc:223] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [CANDIDATE]: Term 1 election: Requesting vote from peer a48b3d8066e2410bb967d2167f2dbdb4 I0628 23:55:37.605262 32192 raft_consensus.cc:2019] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:37.606534 32427 raft_consensus.cc:2019] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:37.609439 32192 raft_consensus.cc:1677] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1edad4aefcff40b38e3ae1170d2d724c in term 1. I0628 23:55:37.610231 32427 raft_consensus.cc:1677] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1edad4aefcff40b38e3ae1170d2d724c in term 1. I0628 23:55:37.610498 32246 leader_election.cc:347] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [CANDIDATE]: Term 1 election: Vote granted by peer cae3c5da9948493a92c33eee962172af I0628 23:55:37.610816 32246 leader_election.cc:248] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:37.611507 32246 leader_election.cc:347] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [CANDIDATE]: Term 1 election: Vote granted by peer a48b3d8066e2410bb967d2167f2dbdb4 I0628 23:55:37.611533 32501 raft_consensus.cc:1859] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:37.611186 27213 cluster_itest_util.cc:187] Not converged past 1 yet: 0.0 0.0 0.0 I0628 23:55:37.613174 32501 raft_consensus.cc:464] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 1 LEADER]: Becoming Leader. State: Replica: 1edad4aefcff40b38e3ae1170d2d724c, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:37.613894 32501 consensus_queue.cc:145] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "cae3c5da9948493a92c33eee962172af" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 42671 } } peers { permanent_uuid: "1edad4aefcff40b38e3ae1170d2d724c" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 48364 } } peers { permanent_uuid: "a48b3d8066e2410bb967d2167f2dbdb4" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 34011 } } I0628 23:55:37.640081 32085 catalog_manager.cc:1673] Tablet: 4815feb633554bff9c217d459d84e0a4 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "1edad4aefcff40b38e3ae1170d2d724c" config { opid_index: -1 peers { permanent_uuid: "cae3c5da9948493a92c33eee962172af" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 42671 } } peers { permanent_uuid: "1edad4aefcff40b38e3ae1170d2d724c" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 48364 } } peers { permanent_uuid: "a48b3d8066e2410bb967d2167f2dbdb4" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 34011 } } } W0628 23:55:37.640874 32507 raft_consensus_state.cc:541] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:55:37.716519 27213 cluster_itest_util.cc:187] Not converged past 1 yet: 0.0 1.1 0.0 I0628 23:55:37.921975 27213 cluster_itest_util.cc:187] Not converged past 1 yet: 0.0 1.1 0.0 I0628 23:55:38.119673 32192 raft_consensus.cc:869] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af [term 1 FOLLOWER]: Refusing update from remote peer 1edad4aefcff40b38e3ae1170d2d724c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0628 23:55:38.120730 32427 raft_consensus.cc:869] T 4815feb633554bff9c217d459d84e0a4 P a48b3d8066e2410bb967d2167f2dbdb4 [term 1 FOLLOWER]: Refusing update from remote peer 1edad4aefcff40b38e3ae1170d2d724c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0628 23:55:38.121659 32501 consensus_queue.cc:578] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [LEADER]: Connected to new peer: Peer: cae3c5da9948493a92c33eee962172af, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:55:38.122920 32508 consensus_queue.cc:578] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [LEADER]: Connected to new peer: Peer: a48b3d8066e2410bb967d2167f2dbdb4, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:55:38.161797 32511 raft_consensus_state.cc:534] T 4815feb633554bff9c217d459d84e0a4 P 1edad4aefcff40b38e3ae1170d2d724c [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32111==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32239==WARNING: external symbolizer didn't start up correctly! ==32239==WARNING: Can't write to symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32111==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32111==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32239==WARNING: Can't read from symbolizer at fd 29 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32239==WARNING: Can't read from symbolizer at fd 29 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32111==WARNING: Can't read from symbolizer at fd 27 ==32239==WARNING: Can't read from symbolizer at fd 29 ==32111==WARNING: Failed to use and restart external symbolizer! ==32239==WARNING: Failed to use and restart external symbolizer! ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7d500003fd38 by thread T123: WARNING: ThreadSanitizer: data race (pid=32239) #0 (libtablet.so+0x0000001fb20c) Atomic write of size 4 at 0x7d700006f8f0 by thread T130: #1 (libtablet.so+0x0000001fad02) #0 (kudu-tserver+0x000000466be0) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023d1ea) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #16 (libconsensus.so+0x0000000c5643) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #17 (libconsensus.so+0x0000000c5401) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #18 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) #19 (libkudu_util.so+0x000000226bcd) Previous read of size 4 at 0x7d700006f8f0 by thread T129: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #20 (libkudu_util.so+0x000000224fcf) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #21 (libkudu_util.so+0x00000022941e) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #22 (libkudu_util.so+0x00000022937b) #9 (libtablet.so+0x00000023d2c6) #23 (libkudu_util.so+0x000000229303) #10 (libtablet.so+0x00000023d255) #24 (libkudu_util.so+0x000000229101) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #25 (libkrpc.so+0x0000000cce71) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #26 (libkudu_util.so+0x00000021b55f) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #27 (kudu-tserver+0x000000421829) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) Previous read of size 8 at 0x7d500003fd38 by thread T124: #22 (libkudu_util.so+0x000000229303) #0 (libtablet.so+0x0000001fb19b) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #1 (libtablet.so+0x0000001fad02) #25 (libkudu_util.so+0x00000021b55f) #2 (libtablet.so+0x000000295cf2) #26 (kudu-tserver+0x000000421829) #3 (libtablet.so+0x00000028c6b7) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #4 (libtablet.so+0x000000289219) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #5 (libtablet.so+0x0000001dcebd) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #6 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #7 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023d2c6) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #12 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #13 (libconsensus.so+0x0000000befdd) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #14 (libconsensus.so+0x0000000c5726) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #15 (libconsensus.so+0x0000000c568e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #16 (libconsensus.so+0x0000000c5643) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) #17 (libconsensus.so+0x0000000c5401) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #18 (libkrpc.so+0x0000000cce71) #0 (kudu-tserver+0x00000044a123) #19 (libkudu_util.so+0x000000226bcd) #1 (libkudu_util.so+0x00000021ade7) #20 (libkudu_util.so+0x000000224fcf) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #21 (libkudu_util.so+0x00000022941e) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #22 (libkudu_util.so+0x00000022937b) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #23 (libkudu_util.so+0x000000229303) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #24 (libkudu_util.so+0x000000229101) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #25 (libkrpc.so+0x0000000cce71) #15 (libconsensus.so+0x0000000f0017) #26 (libkudu_util.so+0x00000021b55f) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #27 (kudu-tserver+0x000000421829) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T117: #21 (libconsensus.so+0x0000000ba558) #0 (kudu-tserver+0x0000004aaf73) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #1 (libtablet.so+0x0000001db43e) #24 (libconsensus.so+0x0000000c5726) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #25 (libconsensus.so+0x0000000c568e) #4 (libtablet.so+0x00000020f54e) #26 (libconsensus.so+0x0000000c5643) #5 (libtserver.so+0x00000014bc0e) #27 (libconsensus.so+0x0000000c5401) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #28 (libkrpc.so+0x0000000cce71) #8 (libtserver.so+0x000000157c63) #29 (libkudu_util.so+0x000000226bcd) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000224fcf) #11 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x00000022941e) #12 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022937b) #13 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x000000229303) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #35 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #36 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #3 (libkudu_util.so+0x000000224304) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #4 (libkudu_util.so+0x000000224902) #7 (libtablet.so+0x00000023ae76) #5 (libkudu_util.so+0x000000224704) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #6 (libkudu_util.so+0x000000224632) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #7 (libtablet.so+0x00000023ae76) #12 (libmaster_rpc.so+0x000000011c31) #8 (libtablet.so+0x00000023908d) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #9 (libtablet.so+0x000000234615) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #10 (libtablet.so+0x000000234508) #19 (libconsensus_proto.so+0x00000006d18e) #11 (libtablet.so+0x00000023444a) #20 (libkrpc.so+0x000000121d48) #12 (libmaster_rpc.so+0x000000011c31) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #13 (libconsensus.so+0x000000096280) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #14 (libconsensus.so+0x0000000f0f2d) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #15 (libconsensus.so+0x0000000f0017) #27 (libkrpc.so+0x0000000cce71) #16 (libconsensus.so+0x0000000d76b1) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) #17 (libconsensus.so+0x0000000d73d0) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #18 (libconsensus.so+0x0000000d79cd) #1 (libkudu_util.so+0x00000021ade7) #19 (libconsensus.so+0x0000000a97f2) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #20 (libconsensus.so+0x0000000ba5e8) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #21 (libconsensus.so+0x0000000ba558) #6 (libkudu_util.so+0x000000224632) #22 (libconsensus.so+0x0000000ba4d3) #7 (libtablet.so+0x00000023ae76) #23 (libconsensus.so+0x0000000befdd) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #24 (libconsensus.so+0x0000000c5726) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #25 (libconsensus.so+0x0000000c568e) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #26 (libconsensus.so+0x0000000c5643) #14 (libconsensus.so+0x0000000f0f2d) #27 (libconsensus.so+0x0000000c5401) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #28 (libkrpc.so+0x0000000cce71) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #29 (libkudu_util.so+0x000000226bcd) #19 (libconsensus_proto.so+0x00000006d18e) #30 (libkudu_util.so+0x000000224fcf) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #31 (libkudu_util.so+0x00000022941e) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #32 (libkudu_util.so+0x00000022937b) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) #35 (libkrpc.so+0x0000000cce71) Thread T117 'tablet-bootstra' (tid=32493, finished) created by thread T50 at: #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) #2 (libkudu_util.so+0x0000002262ae) ================== #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic read of size 8 at 0x7d700006f840 by thread T129: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) #2 (libtablet.so+0x0000001fbd19) ================== #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous write of size 1 at 0x7d700006f844 by thread T130: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) ================== #22 (libtablet.so+0x00000023d1ea) WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7d500003fd38 by thread T124: #23 (libconsensus.so+0x0000000befdd) #0 (libtablet.so+0x0000001fb20c) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #1 (libtablet.so+0x0000001fad02) #26 (libconsensus.so+0x0000000c5643) #2 (libtablet.so+0x000000295cf2) #27 (libconsensus.so+0x0000000c5401) #3 (libtablet.so+0x00000028c6b7) #28 (libkrpc.so+0x0000000cce71) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #29 (libkudu_util.so+0x000000226bcd) #6 (libtablet.so+0x0000001dde5d) #30 (libkudu_util.so+0x000000224fcf) #7 (libtablet.so+0x0000001ddcaf) #31 (libkudu_util.so+0x00000022941e) #8 (libtablet.so+0x0000002438f1) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #9 (libtablet.so+0x00000023b2f5) #34 (libkudu_util.so+0x000000229101) #10 (libtablet.so+0x00000023d2c6) #35 (libkrpc.so+0x0000000cce71) #11 (libtablet.so+0x00000023d255) #36 (libkudu_util.so+0x00000021b55f) #12 (libtablet.so+0x00000023d1ea) #37 (kudu-tserver+0x000000421829) #13 (libconsensus.so+0x0000000befdd) Thread T129 'apply [worker]-' #14 (libconsensus.so+0x0000000c5726) (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #15 (libconsensus.so+0x0000000c568e) #1 (libkudu_util.so+0x00000021ade7) #16 (libconsensus.so+0x0000000c5643) #2 (libkudu_util.so+0x0000002262ae) #17 (libconsensus.so+0x0000000c5401) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #18 (libkrpc.so+0x0000000cce71) #5 (libkudu_util.so+0x000000224704) #19 (libkudu_util.so+0x000000226bcd) #6 (libkudu_util.so+0x000000224632) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #7 (libtablet.so+0x00000023ae76) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #8 (libtablet.so+0x00000023908d) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #9 (libtablet.so+0x000000234615) #26 (libkudu_util.so+0x00000021b55f) #10 (libtablet.so+0x000000234508) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500003fd38 by thread T123: #0 (libtablet.so+0x0000001fb20c) #11 (libtablet.so+0x00000023444a) #1 (libtablet.so+0x0000001fad02) #12 (libmaster_rpc.so+0x000000011c31) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #13 (libconsensus.so+0x000000096280) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #14 (libconsensus.so+0x0000000f0f2d) #6 (libtablet.so+0x0000001dde5d) #15 (libconsensus.so+0x0000000f0017) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #16 (libconsensus.so+0x0000000d76b1) #9 (libtablet.so+0x00000023b2f5) #17 (libconsensus.so+0x0000000d73d0) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #18 (libconsensus.so+0x0000000d79cd) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000a97f2) #14 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000ba5e8) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000ba558) #17 (libconsensus.so+0x0000000c5401) #22 (libconsensus.so+0x0000000ba4d3) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #23 (libconsensus.so+0x0000000befdd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #24 (libconsensus.so+0x0000000c5726) #22 (libkudu_util.so+0x00000022937b) #25 (libconsensus.so+0x0000000c568e) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #26 (libconsensus.so+0x0000000c5643) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (libconsensus.so+0x0000000c5401) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #28 (libkrpc.so+0x0000000cce71) #0 (kudu-tserver+0x0000004480c7) #29 (libkudu_util.so+0x000000226bcd) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #30 (libkudu_util.so+0x000000224fcf) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #31 (libkudu_util.so+0x00000022941e) #5 (libtablet.so+0x0000001fad02) #32 (libkudu_util.so+0x00000022937b) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #33 (libkudu_util.so+0x000000229303) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #34 (libkudu_util.so+0x000000229101) #10 (libtablet.so+0x0000001dde5d) #35 (libkrpc.so+0x0000000cce71) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #36 (libkudu_util.so+0x00000021b55f) #13 (libtablet.so+0x00000023b2f5) #37 (kudu-tserver+0x000000421829) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #16 (libtablet.so+0x00000023d1ea) #0 (kudu-tserver+0x00000044a123) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #1 (libkudu_util.so+0x00000021ade7) #19 (libconsensus.so+0x0000000c568e) #2 (libkudu_util.so+0x0000002262ae) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #3 (libkudu_util.so+0x000000224304) #22 (libkrpc.so+0x0000000cce71) #4 (libkudu_util.so+0x000000224902) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #5 (libkudu_util.so+0x000000224704) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #6 (libkudu_util.so+0x000000224632) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #7 (libtablet.so+0x00000023ae76) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #8 (libtablet.so+0x00000023908d) #31 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T117: #9 (libtablet.so+0x000000234615) #0 (kudu-tserver+0x0000004aaf73) #10 (libtablet.so+0x000000234508) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #11 (libtablet.so+0x00000023444a) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #12 (libmaster_rpc.so+0x000000011c31) #5 (libtserver.so+0x00000014bc0e) #13 (libconsensus.so+0x000000096280) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #14 (libconsensus.so+0x0000000f0f2d) #8 (libtserver.so+0x000000157c63) #15 (libconsensus.so+0x0000000f0017) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #16 (libconsensus.so+0x0000000d76b1) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #17 (libconsensus.so+0x0000000d73d0) #13 (libkudu_util.so+0x00000022941e) #18 (libconsensus.so+0x0000000d79cd) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #19 (libconsensus.so+0x0000000a97f2) #16 (libkudu_util.so+0x000000229101) #20 (libconsensus.so+0x0000000ba5e8) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #21 (libconsensus.so+0x0000000ba558) #19 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' #22 (libconsensus.so+0x0000000ba4d3) (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #23 (libconsensus.so+0x0000000befdd) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #24 (libconsensus.so+0x0000000c5726) #3 (libkudu_util.so+0x000000224304) #25 (libconsensus.so+0x0000000c568e) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #26 (libconsensus.so+0x0000000c5643) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #27 (libconsensus.so+0x0000000c5401) #8 (libtablet.so+0x00000023908d) #28 (libkrpc.so+0x0000000cce71) #9 (libtablet.so+0x000000234615) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #10 (libtablet.so+0x000000234508) #33 (libkudu_util.so+0x000000229303) #11 (libtablet.so+0x00000023444a) #34 (libkudu_util.so+0x000000229101) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #35 (libkrpc.so+0x0000000cce71) #14 (libconsensus.so+0x0000000f0f2d) #36 (libkudu_util.so+0x00000021b55f) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #37 (kudu-tserver+0x000000421829) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x4661ac) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) ================== #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=32493, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic read of size 8 at 0x7d700006f848 by thread T129: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d700006f848 by thread T130: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x4661ac) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Read of size 8 at 0x7d500003fd38 by thread T114 (mutexes: write M3472, write M7922): #0 (libtablet.so+0x0000001f890d) #1 (libtablet.so+0x0000001ebf10) #2 (libtablet.so+0x0000001e5407) #3 (libtablet.so+0x000000228b62) #4 (libtablet.so+0x000000281909) #5 (libtablet.so+0x000000280cc6) #6 (libtablet.so+0x0000002869b6) #7 (libtablet.so+0x00000028691a) #8 (libtablet.so+0x0000002868a3) #9 (libtablet.so+0x000000286679) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x00000021b55f) #12 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500003fd38 by thread T123: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Mutex M3472 (0x7d4400009b48) created at: #0 (kudu-tserver+0x000000434625) #1 (libkudu_util.so+0x0000001eb208) #2 (libtablet.so+0x00000028055a) #3 (libtserver.so+0x000000127713) #4 (kudu-tserver+0x0000004ad348) #5 (kudu-tserver+0x0000004ad1ee) #6 (libc.so.6+0x000000021ec4) Mutex M7922 (0x7d1c0001bee4) created at: #0 (kudu-tserver+0x00000046d6b8) #1 (libtserver.so+0x0000001124f7) #2 (libtserver.so+0x000000112470) #3 (libtserver.so+0x000000112439) #4 (libtserver.so+0x0000001123c8) #5 (libtablet.so+0x000000228b3c) #6 (libtablet.so+0x000000281909) #7 (libtablet.so+0x000000280cc6) #8 (libtablet.so+0x0000002869b6) #9 (libtablet.so+0x00000028691a) #10 (libtablet.so+0x0000002868a3) #11 (libtablet.so+0x000000286679) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x00000021b55f) #14 (kudu-tserver+0x000000421829) Thread T114 'maintenance_sch' (tid=32236, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libtablet.so+0x00000028351d) #3 (libtablet.so+0x000000280b95) #4 (libtserver.so+0x00000012824b) #5 (kudu-tserver+0x0000004ad40c) #6 (kudu-tserver+0x0000004ad1ee) #7 (libc.so.6+0x000000021ec4) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=32493, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1f890d) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Read of size 8 at 0x7d700006f8c0 by thread T129: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d700006f8c0 by thread T130: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) ================== #24 (libconsensus.so+0x0000000c5726) WARNING: ThreadSanitizer: data race (pid=32111) Atomic write of size 4 at 0x7d700005f8f0 by thread T124: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700005f8f0 by thread T123: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #26 (libconsensus.so+0x0000000c5643) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #35 (libkrpc.so+0x0000000cce71) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #36 (libkudu_util.so+0x00000021b55f) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #37 (kudu-tserver+0x000000421829) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb3fb) ================== #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7d700006f808 by thread T131: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f808 by thread T129: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) ================== WARNING: ThreadSanitizer: data race (pid=32111) #27 (libconsensus.so+0x0000000c5401) Atomic write of size 8 at 0x7d700005f818 by thread T123: #28 (libkrpc.so+0x0000000cce71) #0 (kudu-tserver+0x000000466ec0) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #1 (libtablet.so+0x000000227502) #31 (libkudu_util.so+0x00000022941e) #2 (libtablet.so+0x0000002972a5) #32 (libkudu_util.so+0x00000022937b) #3 (libtablet.so+0x000000297240) #33 (libkudu_util.so+0x000000229303) #4 (libtablet.so+0x00000028bc4a) #34 (libkudu_util.so+0x000000229101) #5 (libtablet.so+0x000000298a34) #35 (libkrpc.so+0x0000000cce71) #6 (libtablet.so+0x0000002979d1) #36 (libkudu_util.so+0x00000021b55f) #7 (libtablet.so+0x000000296aeb) #37 (kudu-tserver+0x000000421829) #8 (libtablet.so+0x0000002965d3) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #9 (libtablet.so+0x00000028cafd) #0 (kudu-tserver+0x00000044a123) #10 (libtablet.so+0x00000028936b) #1 (libkudu_util.so+0x00000021ade7) #11 (libtablet.so+0x0000001dcebd) #2 (libkudu_util.so+0x0000002262ae) #12 (libtablet.so+0x0000001dde5d) #3 (libkudu_util.so+0x000000224304) #13 (libtablet.so+0x0000001ddcaf) #4 (libkudu_util.so+0x000000224902) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #5 (libkudu_util.so+0x000000224704) #16 (libtablet.so+0x00000023d2c6) #6 (libkudu_util.so+0x000000224632) #17 (libtablet.so+0x00000023d255) #7 (libtablet.so+0x00000023ae76) #18 (libtablet.so+0x00000023d1ea) #8 (libtablet.so+0x00000023908d) #19 (libconsensus.so+0x0000000befdd) #9 (libtablet.so+0x000000234615) #20 (libconsensus.so+0x0000000c5726) #10 (libtablet.so+0x000000234508) #21 (libconsensus.so+0x0000000c568e) #11 (libtablet.so+0x00000023444a) #22 (libconsensus.so+0x0000000c5643) #12 (libmaster_rpc.so+0x000000011c31) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #13 (libconsensus.so+0x000000096280) #25 (libkudu_util.so+0x000000226bcd) #14 (libconsensus.so+0x0000000f0f2d) #26 (libkudu_util.so+0x000000224fcf) #15 (libconsensus.so+0x0000000f0017) #27 (libkudu_util.so+0x00000022941e) #16 (libconsensus.so+0x0000000d76b1) #28 (libkudu_util.so+0x00000022937b) #17 (libconsensus.so+0x0000000d73d0) #29 (libkudu_util.so+0x000000229303) #18 (libconsensus.so+0x0000000d79cd) #30 (libkudu_util.so+0x000000229101) #19 (libconsensus.so+0x0000000a97f2) #31 (libkrpc.so+0x0000000cce71) #20 (libconsensus.so+0x0000000ba5e8) #32 (libkudu_util.so+0x00000021b55f) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700005f819 by thread T124: #23 (libconsensus.so+0x0000000befdd) #0 (libtablet.so+0x0000001fc02e) #24 (libconsensus.so+0x0000000c5726) #1 (libtablet.so+0x0000001fbf59) #25 (libconsensus.so+0x0000000c568e) #2 (libtablet.so+0x0000001fbbd2) #26 (libconsensus.so+0x0000000c5643) #3 (libtablet.so+0x0000001fb7ea) #27 (libconsensus.so+0x0000000c5401) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #28 (libkrpc.so+0x0000000cce71) #6 (libtablet.so+0x0000001fad7b) #29 (libkudu_util.so+0x000000226bcd) #7 (libtablet.so+0x000000295cf2) #30 (libkudu_util.so+0x000000224fcf) #8 (libtablet.so+0x00000028c6b7) #31 (libkudu_util.so+0x00000022941e) #9 (libtablet.so+0x000000289219) #32 (libkudu_util.so+0x00000022937b) #10 (libtablet.so+0x0000001dcebd) #33 (libkudu_util.so+0x000000229303) #11 (libtablet.so+0x0000001dde5d) #34 (libkudu_util.so+0x000000229101) #12 (libtablet.so+0x0000001ddcaf) #35 (libkrpc.so+0x0000000cce71) #13 (libtablet.so+0x0000002438f1) #36 (libkudu_util.so+0x00000021b55f) #14 (libtablet.so+0x00000023b2f5) #37 (kudu-tserver+0x000000421829) #15 (libtablet.so+0x00000023d2c6) Thread T129 'apply [worker]-' #16 (libtablet.so+0x00000023d255) (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #17 (libtablet.so+0x00000023d1ea) #1 (libkudu_util.so+0x00000021ade7) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #2 (libkudu_util.so+0x0000002262ae) #20 (libconsensus.so+0x0000000c568e) #3 (libkudu_util.so+0x000000224304) #21 (libconsensus.so+0x0000000c5643) #4 (libkudu_util.so+0x000000224902) #22 (libconsensus.so+0x0000000c5401) #5 (libkudu_util.so+0x000000224704) #23 (libkrpc.so+0x0000000cce71) #6 (libkudu_util.so+0x000000224632) #24 (libkudu_util.so+0x000000226bcd) #7 (libtablet.so+0x00000023ae76) #25 (libkudu_util.so+0x000000224fcf) #8 (libtablet.so+0x00000023908d) #26 (libkudu_util.so+0x00000022941e) #9 (libtablet.so+0x000000234615) #27 (libkudu_util.so+0x00000022937b) #10 (libtablet.so+0x000000234508) #28 (libkudu_util.so+0x000000229303) #11 (libtablet.so+0x00000023444a) #29 (libkudu_util.so+0x000000229101) #12 (libmaster_rpc.so+0x000000011c31) #30 (libkrpc.so+0x0000000cce71) #13 (libconsensus.so+0x000000096280) #31 (libkudu_util.so+0x00000021b55f) #14 (libconsensus.so+0x0000000f0f2d) #32 (kudu-tserver+0x000000421829) #15 (libconsensus.so+0x0000000f0017) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #16 (libconsensus.so+0x0000000d76b1) #1 (libkudu_util.so+0x0000001d048f) #17 (libconsensus.so+0x0000000d73d0) #2 (libkudu_util.so+0x0000001d039a) #18 (libconsensus.so+0x0000000d79cd) #3 (libkudu_util.so+0x0000001d11a9) #19 (libconsensus.so+0x0000000a97f2) #4 (libkudu_util.so+0x0000001d0e5f) #20 (libconsensus.so+0x0000000ba5e8) #5 (libkudu_util.so+0x0000001cc3ab) #21 (libconsensus.so+0x0000000ba558) #6 (libkudu_util.so+0x0000001cb1d4) #22 (libconsensus.so+0x0000000ba4d3) #7 (libkudu_util.so+0x0000001cba25) #23 (libconsensus.so+0x0000000befdd) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #24 (libconsensus.so+0x0000000c5726) #10 (libtablet.so+0x000000297af3) #25 (libconsensus.so+0x0000000c568e) #11 (libtablet.so+0x000000296aeb) #26 (libconsensus.so+0x0000000c5643) #12 (libtablet.so+0x0000002965d3) #27 (libconsensus.so+0x0000000c5401) #13 (libtablet.so+0x00000028cafd) #28 (libkrpc.so+0x0000000cce71) #14 (libtablet.so+0x00000028936b) #29 (libkudu_util.so+0x000000226bcd) #15 (libtablet.so+0x0000001dcebd) #30 (libkudu_util.so+0x000000224fcf) #16 (libtablet.so+0x0000001dde5d) #31 (libkudu_util.so+0x00000022941e) #17 (libtablet.so+0x0000001ddcaf) #32 (libkudu_util.so+0x00000022937b) #18 (libtablet.so+0x0000002438f1) #33 (libkudu_util.so+0x000000229303) #19 (libtablet.so+0x00000023b2f5) #34 (libkudu_util.so+0x000000229101) #20 (libtablet.so+0x00000023d2c6) #35 (libkrpc.so+0x0000000cce71) #21 (libtablet.so+0x00000023d255) #36 (libkudu_util.so+0x00000021b55f) #22 (libtablet.so+0x00000023d1ea) #37 (kudu-tserver+0x000000421829) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) #25 (libconsensus.so+0x0000000c568e) ================== #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7d500003fd38 by thread T131: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500003fd38 by thread T132: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=32490, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7d700005f838 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700005f839 by thread T123: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7d700005f828 by thread T125: #0 (libtablet.so+0x00000028bc27) ================== WARNING: ThreadSanitizer: data race (pid=32239) Read of size 8 at 0x7d500003fd38 by thread T129: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fae00) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500003fd38 by thread T131: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500003fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=32490, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7d900003d8c8 by thread T131: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900003d8c8 by thread T132: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700005f828 by thread T123: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7d700005f8a8 by thread T125: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700005f8a8 by thread T123: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7d700005f898 by thread T125: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700005f898 by thread T123: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Atomic read of size 8 at 0x7d700005f848 by thread T126: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous write of size 1 at 0x7d700005f849 by thread T123: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32545, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x4661ac) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7d700005f808 by thread T124: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700005f808 by thread T126: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32545, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Read of size 8 at 0x7d700005f8c8 by thread T126: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d700005f8c8 by thread T124: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700005f800 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32545, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb3fb) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Atomic write of size 4 at 0x7d900003d9d8 by thread T124: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003d9d8 by thread T126: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32545, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7db000053660 by thread T130: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7db000053661 by thread T131: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 31872 at 0x7db000050000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 4 at 0x7d900003da20 by thread T129: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003da20 by thread T131: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic read of size 8 at 0x7d900003d958 by thread T131: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d900003d958 by thread T129: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x4661ac) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Read of size 8 at 0x7d900003d9d0 by thread T131: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d900003d9d0 by thread T129: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb3fb) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32356==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32356==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32356==WARNING: Can't read from symbolizer at fd 27 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32356==WARNING: Can't read from symbolizer at fd 27 ==32356==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=32356) Atomic write of size 4 at 0x7d900003dd60 by thread T122: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003dd60 by thread T124: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7f734414e058 by thread T129: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7f734414e05a by thread T132: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f7344141000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7f734414e0d0 by thread T129: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f734414e0d0 by thread T132: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f7344141000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 1 at 0x7d900003d961 by thread T130: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000297dca) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900003d960 by thread T131: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 8 at 0x7d900003dcb0 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900003dcb0 by thread T125: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32548, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Atomic write of size 8 at 0x7d900003dca8 by thread T123: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003dca9 by thread T125: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32548, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 8 at 0x7d900003dd20 by thread T123: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900003dd20 by thread T125: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003c000 allocated by thread T122: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32548, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 8 at 0x7dd000014870 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dd000014870 by thread T124: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 8 at 0x7dd000014868 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000014869 by thread T124: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 8 at 0x7dd0000148e8 by thread T123: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd0000148e8 by thread T124: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Atomic write of size 4 at 0x7dd000014910 by thread T123: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000014910 by thread T124: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 1 at 0x7dd000014859 by thread T125: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dd000014858 by thread T122: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32548, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7dd00001ea50 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dd00001ea51 by thread T124: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32545, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7dd00001ea48 by thread T123: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dd00001ea48 by thread T124: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32545, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Write of size 8 at 0x7dd00001eac0 by thread T123: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd00001eac0 by thread T124: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32545, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7dd000005590 by thread T132: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dd000005591 by thread T131: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7dd000005588 by thread T132: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dd000005588 by thread T131: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7dd000005580 by thread T132: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000005581 by thread T130: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7dd000005578 by thread T132: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000297dca) #9 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x0000002965d3) #11 (libtablet.so+0x00000028cafd) #12 (libtablet.so+0x00000028936b) #13 (libtablet.so+0x0000001dcebd) #14 (libtablet.so+0x0000001dde5d) #15 (libtablet.so+0x0000001ddcaf) #16 (libtablet.so+0x0000002438f1) #17 (libtablet.so+0x00000023b2f5) #18 (libtablet.so+0x00000023d2c6) #19 (libtablet.so+0x00000023d255) #20 (libtablet.so+0x00000023d1ea) #21 (libconsensus.so+0x0000000befdd) #22 (libconsensus.so+0x0000000c5726) #23 (libconsensus.so+0x0000000c568e) #24 (libconsensus.so+0x0000000c5643) #25 (libconsensus.so+0x0000000c5401) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x000000226bcd) #28 (libkudu_util.so+0x000000224fcf) #29 (libkudu_util.so+0x00000022941e) #30 (libkudu_util.so+0x00000022937b) #31 (libkudu_util.so+0x000000229303) #32 (libkudu_util.so+0x000000229101) #33 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x00000021b55f) #35 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dd000005579 by thread T131: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7dd0000055f8 by thread T132: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd0000055f8 by thread T131: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7dd0000055f0 by thread T132: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd0000055f0 by thread T131: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 4 at 0x7dd000005658 by thread T132: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000297dca) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000005658 by thread T131: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 8 at 0x7dd00001e6b0 by thread T124: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dd00001e6b1 by thread T122: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 8 at 0x7dd00001e698 by thread T124: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd00001e699 by thread T122: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Write of size 8 at 0x7dd00001e718 by thread T124: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd00001e718 by thread T122: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic read of size 8 at 0x7dd000005598 by thread T129: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous write of size 1 at 0x7dd00000559c by thread T130: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000297dca) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x4661ac) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7f73441436a0 by thread T130: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f73441436a0 by thread T132: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f7344141000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7dd0000055b0 by thread T130: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000297dca) #9 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x0000002965d3) #11 (libtablet.so+0x00000028cafd) #12 (libtablet.so+0x00000028936b) #13 (libtablet.so+0x0000001dcebd) #14 (libtablet.so+0x0000001dde5d) #15 (libtablet.so+0x0000001ddcaf) #16 (libtablet.so+0x0000002438f1) #17 (libtablet.so+0x00000023b2f5) #18 (libtablet.so+0x00000023d2c6) #19 (libtablet.so+0x00000023d255) #20 (libtablet.so+0x00000023d1ea) #21 (libconsensus.so+0x0000000befdd) #22 (libconsensus.so+0x0000000c5726) #23 (libconsensus.so+0x0000000c568e) #24 (libconsensus.so+0x0000000c5643) #25 (libconsensus.so+0x0000000c5401) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x000000226bcd) #28 (libkudu_util.so+0x000000224fcf) #29 (libkudu_util.so+0x00000022941e) #30 (libkudu_util.so+0x00000022937b) #31 (libkudu_util.so+0x000000229303) #32 (libkudu_util.so+0x000000229101) #33 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x00000021b55f) #35 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd0000055b1 by thread T132: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7f734038faa8 by thread T132: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f734038faa9 by thread T130: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f7340383000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== I0628 23:55:45.480016 32242 tablet.cc:773] T 4815feb633554bff9c217d459d84e0a4 Flush: entering stage 1 (old memrowset already frozen for inserts) I0628 23:55:45.480262 32242 compaction.cc:588] Selected 1 rowsets to compact: I0628 23:55:45.480408 32242 compaction.cc:591] memrowset(current size on disk: ~0 bytes) I0628 23:55:45.480530 32242 tablet.cc:775] T 4815feb633554bff9c217d459d84e0a4 Memstore in-memory size: 2039575 bytes I0628 23:55:45.480810 32242 tablet.cc:1239] T 4815feb633554bff9c217d459d84e0a4 Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6009479763736694784 or (T in {6009479763736694784})}] I0628 23:55:45.492738 32359 tablet.cc:773] T 4815feb633554bff9c217d459d84e0a4 Flush: entering stage 1 (old memrowset already frozen for inserts) I0628 23:55:45.492962 32359 compaction.cc:588] Selected 1 rowsets to compact: I0628 23:55:45.493109 32359 compaction.cc:591] memrowset(current size on disk: ~0 bytes) I0628 23:55:45.493252 32359 tablet.cc:775] T 4815feb633554bff9c217d459d84e0a4 Memstore in-memory size: 2039575 bytes I0628 23:55:45.493644 32359 tablet.cc:1239] T 4815feb633554bff9c217d459d84e0a4 Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6009479763736694784 or (T in {6009479763736694784})}] I0628 23:55:45.506505 32359 multi_column_writer.cc:85] Opened CFile writer for column key[int32 NOT NULL] I0628 23:55:45.513038 32359 multi_column_writer.cc:85] Opened CFile writer for column int_val[int32 NOT NULL] I0628 23:55:45.518873 32124 tablet.cc:773] T 4815feb633554bff9c217d459d84e0a4 Flush: entering stage 1 (old memrowset already frozen for inserts) I0628 23:55:45.519093 32124 compaction.cc:588] Selected 1 rowsets to compact: I0628 23:55:45.519229 32124 compaction.cc:591] memrowset(current size on disk: ~0 bytes) I0628 23:55:45.519351 32124 tablet.cc:775] T 4815feb633554bff9c217d459d84e0a4 Memstore in-memory size: 2039575 bytes I0628 23:55:45.519836 32124 tablet.cc:1239] T 4815feb633554bff9c217d459d84e0a4 Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6009479763736694784 or (T in {6009479763736694784})}] I0628 23:55:45.520711 32359 multi_column_writer.cc:85] Opened CFile writer for column string_val[string NULLABLE] ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7d7000075810 by thread T132: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d7000075811 by thread T131: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d7000075800 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== I0628 23:55:45.551730 32242 multi_column_writer.cc:85] Opened CFile writer for column key[int32 NOT NULL] I0628 23:55:45.556195 32242 multi_column_writer.cc:85] Opened CFile writer for column int_val[int32 NOT NULL] I0628 23:55:45.584239 32242 multi_column_writer.cc:85] Opened CFile writer for column string_val[string NULLABLE] I0628 23:55:45.627802 32124 multi_column_writer.cc:85] Opened CFile writer for column key[int32 NOT NULL] I0628 23:55:45.629225 32124 multi_column_writer.cc:85] Opened CFile writer for column int_val[int32 NOT NULL] I0628 23:55:45.647874 32124 multi_column_writer.cc:85] Opened CFile writer for column string_val[string NULLABLE] ================== WARNING: ThreadSanitizer: data race (pid=32356) Atomic write of size 8 at 0x7fb598d6e140 by thread T122: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000001dcdf9) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7fb598d6e140 by thread T2 (mutexes: write M66221): #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000001fc50a) #2 (libtablet.so+0x0000001fa768) #3 (libtablet.so+0x00000025f9dd) #4 (libtablet.so+0x000000258179) #5 (libtablet.so+0x00000025543d) #6 (libtablet.so+0x0000001df5d2) #7 (libtablet.so+0x0000001deff5) #8 (libtablet.so+0x0000001de8fe) #9 (libtablet.so+0x000000228d53) #10 (libtablet.so+0x000000282726) #11 (libtablet.so+0x00000028833d) #12 (libtablet.so+0x00000028829b) #13 (libtablet.so+0x000000288223) #14 (libtablet.so+0x000000288021) #15 (libkrpc.so+0x0000000cce71) #16 (libkudu_util.so+0x000000226bcd) #17 (libkudu_util.so+0x000000224fcf) #18 (libkudu_util.so+0x00000022941e) #19 (libkudu_util.so+0x00000022937b) #20 (libkudu_util.so+0x000000229303) #21 (libkudu_util.so+0x000000229101) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x00000021b55f) #24 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7fb598d67000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Mutex M66221 (0x7d500003fd58) created at: #0 (kudu-tserver+0x0000004292c3) #1 (libtablet.so+0x0000001e6e36) #2 (libtablet.so+0x0000001fca79) #3 (libtablet.so+0x0000001ebbef) #4 (libtablet.so+0x0000001dea47) #5 (libtablet.so+0x0000001de8c2) #6 (libtablet.so+0x000000228d53) #7 (libtablet.so+0x000000282726) #8 (libtablet.so+0x00000028833d) #9 (libtablet.so+0x00000028829b) #10 (libtablet.so+0x000000288223) #11 (libtablet.so+0x000000288021) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x000000226bcd) #14 (libkudu_util.so+0x000000224fcf) #15 (libkudu_util.so+0x00000022941e) #16 (libkudu_util.so+0x00000022937b) #17 (libkudu_util.so+0x000000229303) #18 (libkudu_util.so+0x000000229101) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x00000021b55f) #21 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32359, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32356) Atomic write of size 8 at 0x7fb598da0900 by thread T122: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000001dcdf9) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7fb598da0900 by thread T2: [failed to restore the stack] Location is heap block of size 509952 at 0x7fb598d67000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T122 'apply [worker]-' (tid=32529, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32359, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7f73403c2d00 by thread T131: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000001dcdf9) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f73403c2d00 by thread T2 (mutexes: write M69212): #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000001fc50a) #2 (libtablet.so+0x0000001fa768) #3 (libtablet.so+0x00000025f9dd) #4 (libtablet.so+0x000000258179) #5 (libtablet.so+0x00000025543d) #6 (libtablet.so+0x0000001df5d2) #7 (libtablet.so+0x0000001deff5) #8 (libtablet.so+0x0000001de8fe) #9 (libtablet.so+0x000000228d53) #10 (libtablet.so+0x000000282726) #11 (libtablet.so+0x00000028833d) #12 (libtablet.so+0x00000028829b) #13 (libtablet.so+0x000000288223) #14 (libtablet.so+0x000000288021) #15 (libkrpc.so+0x0000000cce71) #16 (libkudu_util.so+0x000000226bcd) #17 (libkudu_util.so+0x000000224fcf) #18 (libkudu_util.so+0x00000022941e) #19 (libkudu_util.so+0x00000022937b) #20 (libkudu_util.so+0x000000229303) #21 (libkudu_util.so+0x000000229101) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x00000021b55f) #24 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f7340383000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Mutex M69212 (0x7d500003fd58) created at: #0 (kudu-tserver+0x0000004292c3) #1 (libtablet.so+0x0000001e6e36) #2 (libtablet.so+0x0000001fca79) #3 (libtablet.so+0x0000001ebbef) #4 (libtablet.so+0x0000001dea47) #5 (libtablet.so+0x0000001de8c2) #6 (libtablet.so+0x000000228d53) #7 (libtablet.so+0x000000282726) #8 (libtablet.so+0x00000028833d) #9 (libtablet.so+0x00000028829b) #10 (libtablet.so+0x000000288223) #11 (libtablet.so+0x000000288021) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x000000226bcd) #14 (libkudu_util.so+0x000000224fcf) #15 (libkudu_util.so+0x00000022941e) #16 (libkudu_util.so+0x00000022937b) #17 (libkudu_util.so+0x000000229303) #18 (libkudu_util.so+0x000000229101) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x00000021b55f) #21 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32242, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Atomic write of size 8 at 0x7fd360684198 by thread T124: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000001dcdf9) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7fd360684198 by thread T2 (mutexes: write M66214): #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000001fc50a) #2 (libtablet.so+0x0000001fa768) #3 (libtablet.so+0x00000025f9dd) #4 (libtablet.so+0x000000258179) #5 (libtablet.so+0x00000025543d) #6 (libtablet.so+0x0000001df5d2) #7 (libtablet.so+0x0000001deff5) #8 (libtablet.so+0x0000001de8fe) #9 (libtablet.so+0x000000228d53) #10 (libtablet.so+0x000000282726) #11 (libtablet.so+0x00000028833d) #12 (libtablet.so+0x00000028829b) #13 (libtablet.so+0x000000288223) #14 (libtablet.so+0x000000288021) #15 (libkrpc.so+0x0000000cce71) #16 (libkudu_util.so+0x000000226bcd) #17 (libkudu_util.so+0x000000224fcf) #18 (libkudu_util.so+0x00000022941e) #19 (libkudu_util.so+0x00000022937b) #20 (libkudu_util.so+0x000000229303) #21 (libkudu_util.so+0x000000229101) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x00000021b55f) #24 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7fd360679000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Mutex M66214 (0x7d500003fd58) created at: #0 (kudu-tserver+0x0000004292c3) #1 (libtablet.so+0x0000001e6e36) #2 (libtablet.so+0x0000001fca79) #3 (libtablet.so+0x0000001ebbef) #4 (libtablet.so+0x0000001dea47) #5 (libtablet.so+0x0000001de8c2) #6 (libtablet.so+0x000000228d53) #7 (libtablet.so+0x000000282726) #8 (libtablet.so+0x00000028833d) #9 (libtablet.so+0x00000028829b) #10 (libtablet.so+0x000000288223) #11 (libtablet.so+0x000000288021) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x000000226bcd) #14 (libkudu_util.so+0x000000224fcf) #15 (libkudu_util.so+0x00000022941e) #16 (libkudu_util.so+0x00000022937b) #17 (libkudu_util.so+0x000000229303) #18 (libkudu_util.so+0x000000229101) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x00000021b55f) #21 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32530, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32124, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Atomic write of size 8 at 0x7fd360683d98 by thread T123: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000001dcdf9) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7fd360683d98 by thread T2: [failed to restore the stack] Location is heap block of size 254976 at 0x7fd360679000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32124, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== I0628 23:55:50.618727 32242 tablet.cc:1319] T 4815feb633554bff9c217d459d84e0a4 Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0628 23:55:50.632279 32124 tablet.cc:1319] T 4815feb633554bff9c217d459d84e0a4 Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0628 23:55:50.633316 32242 tablet.cc:1374] T 4815feb633554bff9c217d459d84e0a4 Flush Phase 2: carrying over any updates which arrived during Phase 1 I0628 23:55:50.633522 32242 tablet.cc:1376] T 4815feb633554bff9c217d459d84e0a4 Phase 2 snapshot: MvccSnapshot[committed={T|T < 6009479784763113472 or (T in {6009479784763113472})}] I0628 23:55:50.640950 32124 tablet.cc:1374] T 4815feb633554bff9c217d459d84e0a4 Flush Phase 2: carrying over any updates which arrived during Phase 1 I0628 23:55:50.641212 32124 tablet.cc:1376] T 4815feb633554bff9c217d459d84e0a4 Phase 2 snapshot: MvccSnapshot[committed={T|T < 6009479784531722240 or (T in {6009479784531722240})}] ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7d400003ec00 by thread T132: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000002a89bf) #8 (libtablet.so+0x0000001dcdf9) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d400003ec00 by thread T2 (mutexes: write M69212): #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000001faab2) #2 (libtablet.so+0x0000001fa696) #3 (libtablet.so+0x00000028d128) #4 (libtablet.so+0x000000289de2) #5 (libtablet.so+0x000000289c1e) #6 (libtablet.so+0x000000253549) #7 (libtablet.so+0x00000025347a) #8 (libtablet.so+0x000000289ee3) #9 (libtablet.so+0x0000002538b5) #10 (libtablet.so+0x0000001dff11) #11 (libtablet.so+0x0000001deff5) #12 (libtablet.so+0x0000001de8fe) #13 (libtablet.so+0x000000228d53) #14 (libtablet.so+0x000000282726) #15 (libtablet.so+0x00000028833d) #16 (libtablet.so+0x00000028829b) #17 (libtablet.so+0x000000288223) #18 (libtablet.so+0x000000288021) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Location is heap block of size 249 at 0x7d400003ec00 allocated by thread T116: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x00000028c022) #11 (libtablet.so+0x0000002889d4) #12 (libtablet.so+0x0000001db496) #13 (libtablet.so+0x000000210662) #14 (libtablet.so+0x00000020f9b7) #15 (libtablet.so+0x00000020f54e) #16 (libtserver.so+0x00000014bc0e) #17 (libtserver.so+0x000000157d97) #18 (libtserver.so+0x000000157ce3) #19 (libtserver.so+0x000000157c63) #20 (libtserver.so+0x000000157a21) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Mutex M69212 (0x7d500003fd58) created at: #0 (kudu-tserver+0x0000004292c3) #1 (libtablet.so+0x0000001e6e36) #2 (libtablet.so+0x0000001fca79) #3 (libtablet.so+0x0000001ebbef) #4 (libtablet.so+0x0000001dea47) #5 (libtablet.so+0x0000001de8c2) #6 (libtablet.so+0x000000228d53) #7 (libtablet.so+0x000000282726) #8 (libtablet.so+0x00000028833d) #9 (libtablet.so+0x00000028829b) #10 (libtablet.so+0x000000288223) #11 (libtablet.so+0x000000288021) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x000000226bcd) #14 (libkudu_util.so+0x000000224fcf) #15 (libkudu_util.so+0x00000022941e) #16 (libkudu_util.so+0x00000022937b) #17 (libkudu_util.so+0x000000229303) #18 (libkudu_util.so+0x000000229101) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x00000021b55f) #21 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32242, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T116 'tablet-bootstra' (tid=32490, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Atomic write of size 8 at 0x7d400003ec00 by thread T123: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000002a89bf) #8 (libtablet.so+0x0000001dcdf9) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d400003ec00 by thread T2 (mutexes: write M66214): #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000001faab2) #2 (libtablet.so+0x0000001fa696) #3 (libtablet.so+0x00000028d128) #4 (libtablet.so+0x000000289de2) #5 (libtablet.so+0x000000289c1e) #6 (libtablet.so+0x000000253549) #7 (libtablet.so+0x00000025347a) #8 (libtablet.so+0x000000289ee3) #9 (libtablet.so+0x0000002538b5) #10 (libtablet.so+0x0000001dff11) #11 (libtablet.so+0x0000001deff5) #12 (libtablet.so+0x0000001de8fe) #13 (libtablet.so+0x000000228d53) #14 (libtablet.so+0x000000282726) #15 (libtablet.so+0x00000028833d) #16 (libtablet.so+0x00000028829b) #17 (libtablet.so+0x000000288223) #18 (libtablet.so+0x000000288021) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Location is heap block of size 249 at 0x7d400003ec00 allocated by thread T117: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x00000028c022) #11 (libtablet.so+0x0000002889d4) #12 (libtablet.so+0x0000001db496) #13 (libtablet.so+0x000000210662) #14 (libtablet.so+0x00000020f9b7) #15 (libtablet.so+0x00000020f54e) #16 (libtserver.so+0x00000014bc0e) #17 (libtserver.so+0x000000157d97) #18 (libtserver.so+0x000000157ce3) #19 (libtserver.so+0x000000157c63) #20 (libtserver.so+0x000000157a21) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Mutex M66214 (0x7d500003fd58) created at: #0 (kudu-tserver+0x0000004292c3) #1 (libtablet.so+0x0000001e6e36) #2 (libtablet.so+0x0000001fca79) #3 (libtablet.so+0x0000001ebbef) #4 (libtablet.so+0x0000001dea47) #5 (libtablet.so+0x0000001de8c2) #6 (libtablet.so+0x000000228d53) #7 (libtablet.so+0x000000282726) #8 (libtablet.so+0x00000028833d) #9 (libtablet.so+0x00000028829b) #10 (libtablet.so+0x000000288223) #11 (libtablet.so+0x000000288021) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x000000226bcd) #14 (libkudu_util.so+0x000000224fcf) #15 (libkudu_util.so+0x00000022941e) #16 (libkudu_util.so+0x00000022937b) #17 (libkudu_util.so+0x000000229303) #18 (libkudu_util.so+0x000000229101) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x00000021b55f) #21 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32124, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T117 'tablet-bootstra' (tid=32493, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Atomic write of size 8 at 0x7fd36069a210 by thread T123: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000002a89bf) #8 (libtablet.so+0x0000001dcdf9) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7fd36069a210 by thread T2: [failed to restore the stack] Location is heap block of size 254976 at 0x7fd360679000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32527, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32124, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32111) Atomic write of size 8 at 0x7fd3606b4b58 by thread T126: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000002a89bf) #8 (libtablet.so+0x0000001dcdf9) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7fd3606b4b58 by thread T2: [failed to restore the stack] Location is heap block of size 254976 at 0x7fd360679000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32545, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32124, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T125 'apply [worker]-' (tid=32535, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7f7340396bc0 by thread T129: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000002a89bf) #8 (libtablet.so+0x0000001dcdf9) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f7340396bc0 by thread T2: [failed to restore the stack] Location is heap block of size 509952 at 0x7f7340383000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32242, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T131 'apply [worker]-' (tid=32528, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Write of size 8 at 0x7d900004f460 by thread T132: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900004f460 by thread T130: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T132: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=32526, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) ================== ================== WARNING: ThreadSanitizer: data race (pid=32239) Atomic write of size 8 at 0x7dc00001d8d0 by thread T132: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dc00001d8d1 by thread T129: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000010000 allocated by thread T132: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=32536, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=32525, running) created by thread T126 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== I0628 23:55:51.993914 32359 tablet.cc:1319] T 4815feb633554bff9c217d459d84e0a4 Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0628 23:55:52.003945 32359 tablet.cc:1374] T 4815feb633554bff9c217d459d84e0a4 Flush Phase 2: carrying over any updates which arrived during Phase 1 I0628 23:55:52.004133 32359 tablet.cc:1376] T 4815feb633554bff9c217d459d84e0a4 Phase 2 snapshot: MvccSnapshot[committed={T|T < 6009479789077577728 or (T in {6009479789077577728})}] ================== WARNING: ThreadSanitizer: data race (pid=32356) Atomic write of size 8 at 0x7fb598dcea28 by thread T124: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000002a89bf) #8 (libtablet.so+0x0000001dcdf9) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7fb598dcea28 by thread T2 (mutexes: write M66221): #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000001fc50a) #2 (libtablet.so+0x0000001fa768) #3 (libtablet.so+0x00000025f9dd) #4 (libtablet.so+0x000000258179) #5 (libtablet.so+0x00000025636d) #6 (libtablet.so+0x0000001dff78) #7 (libtablet.so+0x0000001deff5) #8 (libtablet.so+0x0000001de8fe) #9 (libtablet.so+0x000000228d53) #10 (libtablet.so+0x000000282726) #11 (libtablet.so+0x00000028833d) #12 (libtablet.so+0x00000028829b) #13 (libtablet.so+0x000000288223) #14 (libtablet.so+0x000000288021) #15 (libkrpc.so+0x0000000cce71) #16 (libkudu_util.so+0x000000226bcd) #17 (libkudu_util.so+0x000000224fcf) #18 (libkudu_util.so+0x00000022941e) #19 (libkudu_util.so+0x00000022937b) #20 (libkudu_util.so+0x000000229303) #21 (libkudu_util.so+0x000000229101) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x00000021b55f) #24 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7fb598d67000 allocated by thread T123: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Mutex M66221 (0x7d500003fd58) created at: #0 (kudu-tserver+0x0000004292c3) #1 (libtablet.so+0x0000001e6e36) #2 (libtablet.so+0x0000001fca79) #3 (libtablet.so+0x0000001ebbef) #4 (libtablet.so+0x0000001dea47) #5 (libtablet.so+0x0000001de8c2) #6 (libtablet.so+0x000000228d53) #7 (libtablet.so+0x000000282726) #8 (libtablet.so+0x00000028833d) #9 (libtablet.so+0x00000028829b) #10 (libtablet.so+0x000000288223) #11 (libtablet.so+0x000000288021) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x000000226bcd) #14 (libkudu_util.so+0x000000224fcf) #15 (libkudu_util.so+0x00000022941e) #16 (libkudu_util.so+0x00000022937b) #17 (libkudu_util.so+0x000000229303) #18 (libkudu_util.so+0x000000229101) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x00000021b55f) #21 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32359, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== I0628 23:55:52.090376 32124 tablet.cc:1417] T 4815feb633554bff9c217d459d84e0a4 Flush successful on 11100 rows (269976 bytes) I0628 23:55:52.091298 32124 maintenance_manager.cc:357] Time spent running FlushMRSOp(4815feb633554bff9c217d459d84e0a4): real 6.574s user 2.688s sys 0.024s ================== WARNING: ThreadSanitizer: data race (pid=32356) Atomic write of size 8 at 0x7d400003ec00 by thread T125: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289b02) #7 (libtablet.so+0x0000002a89bf) #8 (libtablet.so+0x0000001dcdf9) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d400003ec00 by thread T2 (mutexes: write M66221): #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000001faab2) #2 (libtablet.so+0x0000001fa696) #3 (libtablet.so+0x00000028d128) #4 (libtablet.so+0x000000289de2) #5 (libtablet.so+0x000000289c1e) #6 (libtablet.so+0x000000253549) #7 (libtablet.so+0x00000025347a) #8 (libtablet.so+0x000000289ee3) #9 (libtablet.so+0x0000002538b5) #10 (libtablet.so+0x0000001dff11) #11 (libtablet.so+0x0000001deff5) #12 (libtablet.so+0x0000001de8fe) #13 (libtablet.so+0x000000228d53) #14 (libtablet.so+0x000000282726) #15 (libtablet.so+0x00000028833d) #16 (libtablet.so+0x00000028829b) #17 (libtablet.so+0x000000288223) #18 (libtablet.so+0x000000288021) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Location is heap block of size 249 at 0x7d400003ec00 allocated by thread T116: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x00000028c022) #11 (libtablet.so+0x0000002889d4) #12 (libtablet.so+0x0000001db496) #13 (libtablet.so+0x000000210662) #14 (libtablet.so+0x00000020f9b7) #15 (libtablet.so+0x00000020f54e) #16 (libtserver.so+0x00000014bc0e) #17 (libtserver.so+0x000000157d97) #18 (libtserver.so+0x000000157ce3) #19 (libtserver.so+0x000000157c63) #20 (libtserver.so+0x000000157a21) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Mutex M66221 (0x7d500003fd58) created at: #0 (kudu-tserver+0x0000004292c3) #1 (libtablet.so+0x0000001e6e36) #2 (libtablet.so+0x0000001fca79) #3 (libtablet.so+0x0000001ebbef) #4 (libtablet.so+0x0000001dea47) #5 (libtablet.so+0x0000001de8c2) #6 (libtablet.so+0x000000228d53) #7 (libtablet.so+0x000000282726) #8 (libtablet.so+0x00000028833d) #9 (libtablet.so+0x00000028829b) #10 (libtablet.so+0x000000288223) #11 (libtablet.so+0x000000288021) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x000000226bcd) #14 (libkudu_util.so+0x000000224fcf) #15 (libkudu_util.so+0x00000022941e) #16 (libkudu_util.so+0x00000022937b) #17 (libkudu_util.so+0x000000229303) #18 (libkudu_util.so+0x000000229101) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x00000021b55f) #21 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32548, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T2 'MaintenanceMgr ' (tid=32359, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000223b4a) #5 (libkudu_util.so+0x000000223a82) #6 (libtablet.so+0x0000002806c5) #7 (libtserver.so+0x000000127713) #8 (kudu-tserver+0x0000004ad348) #9 (kudu-tserver+0x0000004ad1ee) #10 (libc.so.6+0x000000021ec4) Thread T116 'tablet-bootstra' (tid=32492, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x46da28) ================== I0628 23:55:52.185080 32242 tablet.cc:1417] T 4815feb633554bff9c217d459d84e0a4 Flush successful on 11100 rows (269976 bytes) I0628 23:55:52.189359 32242 maintenance_manager.cc:357] Time spent running FlushMRSOp(4815feb633554bff9c217d459d84e0a4): real 6.711s user 2.676s sys 0.132s I0628 23:55:52.263859 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 32239 I0628 23:55:52.322196 32172 tablet_service.cc:653] Processing DeleteTablet for tablet 4815feb633554bff9c217d459d84e0a4 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:49228 I0628 23:55:52.322710 32172 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 4815feb633554bff9c217d459d84e0a4 I0628 23:55:52.323221 32172 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(4815feb633554bff9c217d459d84e0a4) I0628 23:55:52.323525 32172 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(4815feb633554bff9c217d459d84e0a4) I0628 23:55:52.323765 32172 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(4815feb633554bff9c217d459d84e0a4) I0628 23:55:52.324061 32172 maintenance_manager.cc:180] Unregistered op FlushMRSOp(4815feb633554bff9c217d459d84e0a4) I0628 23:55:52.324291 32172 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(4815feb633554bff9c217d459d84e0a4) I0628 23:55:52.324534 32172 maintenance_manager.cc:180] Unregistered op LogGCOp(4815feb633554bff9c217d459d84e0a4) I0628 23:55:52.324733 32172 raft_consensus.cc:1495] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:55:52.325168 32172 raft_consensus.cc:1511] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:55:52.338953 32172 ts_tablet_manager.cc:955] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:55:52.349932 32172 ts_tablet_manager.cc:965] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: Tablet deleted. Last logged OpId: 1.319 I0628 23:55:52.351469 32172 tablet_bootstrap.cc:393] T 4815feb633554bff9c217d459d84e0a4 P cae3c5da9948493a92c33eee962172af: Deleted tablet blocks from disk I0628 23:55:52.365361 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 32067 ================== WARNING: ThreadSanitizer: data race (pid=32356) Atomic write of size 8 at 0x7fb592f48350 by thread T124: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7fb592f48351 by thread T123: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7fb592f2f000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T124 'apply [worker]-' (tid=32546, running) created by thread T121 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T123 'apply [worker]-' (tid=32532, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32548, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== I0628 23:55:52.402664 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 32111 I0628 23:55:52.436619 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 32356 [ OK ] DeleteTableTest.TestOrphanedBlocksClearedOnDelete (16550 ms) [ RUN ] DeleteTableTest.TestFDsNotLeakedOnTabletTombstone I0628 23:55:52.480376 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:52.601161 32562 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:52.601714 32562 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:52.615954 32562 master_main.cc:58] Initializing master server... I0628 23:55:52.618338 32562 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 162351 I0628 23:55:52.625789 32562 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:55:52.626003 32562 server_base.cc:165] Creating new FS layout I0628 23:55:52.639734 32562 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/master/instance: uuid: "8788458e8ebf4aa8b78018b20496f435" format_stamp: "Formatted at 2016-06-28 23:55:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:52.652847 32562 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/master uuid: "8788458e8ebf4aa8b78018b20496f435" format_stamp: "Formatted at 2016-06-28 23:55:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:52.687444 32562 master_main.cc:61] Starting Master server... I0628 23:55:52.722551 32562 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:37752 I0628 23:55:52.724249 32562 webserver.cc:122] Starting webserver on localhost:0 I0628 23:55:52.724376 32562 webserver.cc:133] Document root disabled I0628 23:55:52.725167 32562 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:55:52.726474 32562 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:52389/ I0628 23:55:52.727857 32562 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:55:52.738487 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 32562 I0628 23:55:52.739640 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:37752 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:52.756394 32605 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435: Bootstrap starting. I0628 23:55:52.760253 32605 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:55:52.760893 32605 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435: No blocks or log segments found. Creating new log. I0628 23:55:52.762912 32605 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:52.788614 32605 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435: No bootstrap required, opened a new log I0628 23:55:52.804934 32605 raft_consensus.cc:269] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "8788458e8ebf4aa8b78018b20496f435" member_type: VOTER } I0628 23:55:52.805986 32605 raft_consensus.cc:301] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:52.806253 32605 raft_consensus.cc:499] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8788458e8ebf4aa8b78018b20496f435, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:52.806668 32605 consensus_queue.cc:162] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:52.806948 32605 raft_consensus.cc:314] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:55:52.807096 32605 raft_consensus.cc:370] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:52.807212 32605 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:52.809016 32605 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.621s I0628 23:55:52.809201 32605 raft_consensus.cc:386] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "8788458e8ebf4aa8b78018b20496f435" member_type: VOTER } I0628 23:55:52.811336 32605 leader_election.cc:248] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:52.820824 32610 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.595s I0628 23:55:52.821266 32610 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:52.836531 32610 raft_consensus.cc:464] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 1 LEADER]: Becoming Leader. State: Replica: 8788458e8ebf4aa8b78018b20496f435, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:52.837453 32610 consensus_queue.cc:145] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8788458e8ebf4aa8b78018b20496f435" member_type: VOTER } I0628 23:55:52.857981 32612 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8788458e8ebf4aa8b78018b20496f435. Latest consensus state: current_term: 1 leader_uuid: "8788458e8ebf4aa8b78018b20496f435" config { opid_index: -1 peers { permanent_uuid: "8788458e8ebf4aa8b78018b20496f435" member_type: VOTER } } I0628 23:55:52.858937 32612 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [sys.catalog]: This master's current role is: LEADER I0628 23:55:52.858719 32605 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "8788458e8ebf4aa8b78018b20496f435" config { opid_index: -1 peers { permanent_uuid: "8788458e8ebf4aa8b78018b20496f435" member_type: VOTER } } I0628 23:55:52.859412 32605 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [sys.catalog]: This master's current role is: LEADER I0628 23:55:52.858197 32613 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8788458e8ebf4aa8b78018b20496f435" config { opid_index: -1 peers { permanent_uuid: "8788458e8ebf4aa8b78018b20496f435" member_type: VOTER } } I0628 23:55:52.860452 32613 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [sys.catalog]: This master's current role is: LEADER I0628 23:55:52.865743 32606 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:52.866356 32606 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:52.866638 32605 sys_catalog.cc:311] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:55:52.868120 32614 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:55:52.869508 32615 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 8788458e8ebf4aa8b78018b20496f435 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:55:52.880609 32562 master_main.cc:64] Master server successfully started. I0628 23:55:52.883090 32606 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:52.887465 32606 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 162351 I0628 23:55:52.894387 32606 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:55:52.894613 32606 server_base.cc:165] Creating new FS layout I0628 23:55:52.902912 32606 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "82b5d1288a434f679cf4cb9877d00adb" format_stamp: "Formatted at 2016-06-28 23:55:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:52.914573 32606 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0 uuid: "82b5d1288a434f679cf4cb9877d00adb" format_stamp: "Formatted at 2016-06-28 23:55:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:52.972523 32606 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:53.407907 32606 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:46243 I0628 23:55:53.409435 32606 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:55:53.409544 32606 webserver.cc:133] Document root disabled I0628 23:55:53.411288 32606 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:48103/ I0628 23:55:53.412755 32606 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:53.421654 32606 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:53.423296 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 32606 I0628 23:55:53.433902 32730 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:37752 I0628 23:55:53.434315 32730 heartbeater.cc:319] Registering TS with master... I0628 23:55:53.434689 32730 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:53.436827 32582 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "82b5d1288a434f679cf4cb9877d00adb" instance_seqno: 1467158153386181 } with Master I0628 23:55:53.445616 27213 external_mini_cluster.cc:319] 1 TS(s) registered with Master I0628 23:55:53.473824 32582 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:40620: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:55:53.519593 32667 tablet_service.cc:612] Processing CreateTablet for tablet 7d9f58406a98446db48cb4c59bc42c3c (table=test-workload [id=cb79477f61d6449787016cf9bb3b1917]), partition=range: [(), ()) I0628 23:55:53.535806 32667 ts_tablet_manager.cc:760] Registered tablet 7d9f58406a98446db48cb4c59bc42c3c I0628 23:55:53.537644 32749 ts_tablet_manager.cc:628] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Bootstrapping tablet I0628 23:55:53.542302 32749 tablet_bootstrap.cc:393] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Bootstrap starting. I0628 23:55:53.546066 32749 tablet_bootstrap.cc:553] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:53.546579 32749 tablet_bootstrap.cc:493] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: No blocks or log segments found. Creating new log. I0628 23:55:53.548620 32749 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:53.552809 32749 tablet_bootstrap.cc:393] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: No bootstrap required, opened a new log I0628 23:55:53.553149 32749 ts_tablet_manager.cc:633] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s I0628 23:55:53.559613 32749 raft_consensus.cc:269] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "82b5d1288a434f679cf4cb9877d00adb" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46243 } } I0628 23:55:53.560981 32749 raft_consensus.cc:301] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:53.561275 32749 raft_consensus.cc:499] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b5d1288a434f679cf4cb9877d00adb, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:53.561635 32749 consensus_queue.cc:162] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:53.561910 32749 raft_consensus.cc:314] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:55:53.562088 32749 raft_consensus.cc:370] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:53.562225 32749 raft_consensus.cc:2019] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:53.563964 32749 raft_consensus.cc:1970] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.575s I0628 23:55:53.564183 32749 raft_consensus.cc:386] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "82b5d1288a434f679cf4cb9877d00adb" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46243 } } I0628 23:55:53.566339 32749 leader_election.cc:248] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:53.568197 32754 raft_consensus.cc:1970] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.645s I0628 23:55:53.568549 32754 raft_consensus.cc:1859] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:53.571276 32749 ts_tablet_manager.cc:659] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s I0628 23:55:53.572330 32754 raft_consensus.cc:464] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 1 LEADER]: Becoming Leader. State: Replica: 82b5d1288a434f679cf4cb9877d00adb, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:53.573232 32754 consensus_queue.cc:145] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "82b5d1288a434f679cf4cb9877d00adb" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46243 } } I0628 23:55:53.609159 32582 catalog_manager.cc:1673] Tablet: 7d9f58406a98446db48cb4c59bc42c3c reported consensus state change. New consensus state: current_term: 1 leader_uuid: "82b5d1288a434f679cf4cb9877d00adb" config { opid_index: -1 peers { permanent_uuid: "82b5d1288a434f679cf4cb9877d00adb" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46243 } } } I0628 23:55:53.610312 32757 raft_consensus_state.cc:534] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32606==WARNING: Can't read from symbolizer at fd 26 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32606==WARNING: Can't read from symbolizer at fd 26 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32606==WARNING: external symbolizer didn't start up correctly! ==32606==WARNING: Can't write to symbolizer at fd 29 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==32606==WARNING: Can't read from symbolizer at fd 26 ==32606==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=32606) Atomic write of size 4 at 0x7db000043ae8 by thread T127: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7db000043ae8 by thread T125: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 31872 at 0x7db000040000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=32766, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ================== WARNING: ThreadSanitizer: data race (pid=32606) Read of size 8 at 0x7d400003ec00 by thread T125: #0 (libtablet.so+0x000000296189) #1 (libtablet.so+0x000000295f11) #2 (libtablet.so+0x000000295d63) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous atomic write of size 8 at 0x7d400003ec00 by thread T127: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289219) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Location is heap block of size 249 at 0x7d400003ec00 allocated by thread T116: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x00000028c022) #11 (libtablet.so+0x0000002889d4) #12 (libtablet.so+0x0000001db496) #13 (libtablet.so+0x000000210662) #14 (libtablet.so+0x00000020f9b7) #15 (libtablet.so+0x00000020f54e) #16 (libtserver.so+0x00000014bc0e) #17 (libtserver.so+0x000000157d97) #18 (libtserver.so+0x000000157ce3) #19 (libtserver.so+0x000000157c63) #20 (libtserver.so+0x000000157a21) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=32766, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=32749, running) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x296189) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 4 at 0x7d700006f8f0 by thread T126: #0 (libtablet.so+0x000000299980) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T125: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299980) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d900004fde0 by thread T126: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900004fde1 by thread T128: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d900004fdd8 by thread T126: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900004fdd8 by thread T128: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d900004fdd0 by thread T126: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900004fdd1 by thread T128: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Atomic write of size 8 at 0x7d900004fdc0 by thread T126: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900004fdc1 by thread T127: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=32766, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d900004fe50 by thread T126: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900004fe50 by thread T128: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Atomic write of size 4 at 0x7d900004fea0 by thread T126: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900004fea0 by thread T128: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d700006f840 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f840 by thread T128: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000002960ab) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000298858) #5 (libtablet.so+0x0000002979a4) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d700006f838 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f839 by thread T128: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000002960ab) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000298858) #5 (libtablet.so+0x0000002979a4) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d700006f828 by thread T125: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f829 by thread T128: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000002960ab) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000298858) #5 (libtablet.so+0x0000002979a4) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 1 at 0x7d700006f811 by thread T125: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f810 by thread T127: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb826) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000002960ab) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000298858) #5 (libtablet.so+0x0000002979a4) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=32766, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d700006f8b8 by thread T125: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8b8 by thread T128: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000002960ab) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000298858) #5 (libtablet.so+0x0000002979a4) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T128 'apply [worker]-' (tid=301, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 1 at 0x7da000002c61 by thread T125: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7da000002c60 by thread T127: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 15936 at 0x7da000000000 allocated by thread T126: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=32766, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=32606) Write of size 8 at 0x7d900004fe58 by thread T126: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900004fe58 by thread T127: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T125: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T126 'apply [worker]-' (tid=32765, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T127 'apply [worker]-' (tid=32766, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T125 'apply [worker]-' (tid=32764, running) created by thread T123 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== I0628 23:55:54.214318 32667 tablet_service.cc:653] Processing DeleteTablet for tablet 7d9f58406a98446db48cb4c59bc42c3c with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:46079 I0628 23:55:54.214687 32667 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 7d9f58406a98446db48cb4c59bc42c3c I0628 23:55:54.215076 32667 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(7d9f58406a98446db48cb4c59bc42c3c) I0628 23:55:54.215348 32667 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(7d9f58406a98446db48cb4c59bc42c3c) I0628 23:55:54.215540 32667 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(7d9f58406a98446db48cb4c59bc42c3c) I0628 23:55:54.215751 32667 maintenance_manager.cc:180] Unregistered op FlushMRSOp(7d9f58406a98446db48cb4c59bc42c3c) I0628 23:55:54.215929 32667 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(7d9f58406a98446db48cb4c59bc42c3c) I0628 23:55:54.216109 32667 maintenance_manager.cc:180] Unregistered op LogGCOp(7d9f58406a98446db48cb4c59bc42c3c) I0628 23:55:54.216308 32667 raft_consensus.cc:1495] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 1 LEADER]: Raft consensus shutting down. I0628 23:55:54.216787 32667 raft_consensus.cc:1511] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb [term 1 LEADER]: Raft consensus is shut down! I0628 23:55:54.222823 32667 ts_tablet_manager.cc:955] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:55:54.227262 32667 ts_tablet_manager.cc:965] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Tablet deleted. Last logged OpId: 1.25 I0628 23:55:54.228476 32667 tablet_bootstrap.cc:393] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Deleted tablet blocks from disk I0628 23:55:54.259729 27213 delete_table-test.cc:840] There are 0 open WAL files for pid 32606: I0628 23:55:54.260149 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 32606 I0628 23:55:54.282026 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:46243 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=48103 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:37752 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:54.400192 310 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:54.400745 310 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:54.413697 310 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:54.416556 310 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 163351 I0628 23:55:54.428614 310 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0 uuid: "82b5d1288a434f679cf4cb9877d00adb" format_stamp: "Formatted at 2016-06-28 23:55:52 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:54.464793 310 ts_tablet_manager.cc:604] Loading metadata for tablet 7d9f58406a98446db48cb4c59bc42c3c I0628 23:55:54.470068 310 ts_tablet_manager.cc:936] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_TOMBSTONED I0628 23:55:54.476032 310 ts_tablet_manager.cc:955] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:55:54.482409 310 ts_tablet_manager.cc:965] T 7d9f58406a98446db48cb4c59bc42c3c P 82b5d1288a434f679cf4cb9877d00adb: Tablet deleted. Last logged OpId: 1.25 I0628 23:55:54.483146 310 ts_tablet_manager.cc:760] Registered tablet 7d9f58406a98446db48cb4c59bc42c3c I0628 23:55:54.484143 310 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:54.574301 310 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:46243 I0628 23:55:54.575747 310 webserver.cc:122] Starting webserver on 127.106.77.0:48103 I0628 23:55:54.575851 310 webserver.cc:133] Document root disabled I0628 23:55:54.577447 310 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:48103/ I0628 23:55:54.578675 310 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.DeleteTableTest.TestFDsNotLeakedOnTabletTombstone.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:54.581383 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 310 I0628 23:55:54.590073 310 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:54.597340 424 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:37752 I0628 23:55:54.597720 424 heartbeater.cc:319] Registering TS with master... I0628 23:55:54.598002 424 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:54.600062 32577 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "82b5d1288a434f679cf4cb9877d00adb" instance_seqno: 1467158154555582 } with Master I0628 23:55:54.617166 27213 delete_table-test.cc:840] There are 0 open WAL files for pid 310: I0628 23:55:54.618278 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 32562 I0628 23:55:54.636690 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 310 [ OK ] DeleteTableTest.TestFDsNotLeakedOnTabletTombstone (2178 ms) [----------] 10 tests from DeleteTableTest (71869 ms total) [----------] 3 tests from FaultFlags/DeleteTableDeletedParamTest [ RUN ] FaultFlags/DeleteTableDeletedParamTest.TestRollForwardDelete/0 I0628 23:55:54.658648 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:54.775908 435 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:54.776491 435 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:54.789065 435 master_main.cc:58] Initializing master server... I0628 23:55:54.791425 435 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 163351 I0628 23:55:54.797704 435 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:55:54.797900 435 server_base.cc:165] Creating new FS layout I0628 23:55:54.811203 435 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/master/instance: uuid: "b92976556b684c5fa4a4d8aa36d9cec8" format_stamp: "Formatted at 2016-06-28 23:55:54 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:54.824347 435 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/master uuid: "b92976556b684c5fa4a4d8aa36d9cec8" format_stamp: "Formatted at 2016-06-28 23:55:54 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:54.857087 435 master_main.cc:61] Starting Master server... I0628 23:55:54.892048 435 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:43899 I0628 23:55:54.893604 435 webserver.cc:122] Starting webserver on localhost:0 I0628 23:55:54.893728 435 webserver.cc:133] Document root disabled I0628 23:55:54.894500 435 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:55:54.895762 435 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:60628/ I0628 23:55:54.897253 435 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:55:54.905869 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 435 I0628 23:55:54.906898 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43899 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:54.924873 478 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8: Bootstrap starting. I0628 23:55:54.928740 478 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:55:54.929282 478 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8: No blocks or log segments found. Creating new log. I0628 23:55:54.930850 478 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:54.935814 478 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8: No bootstrap required, opened a new log I0628 23:55:54.941259 478 raft_consensus.cc:269] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" member_type: VOTER } I0628 23:55:54.942198 478 raft_consensus.cc:301] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:54.942409 478 raft_consensus.cc:499] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92976556b684c5fa4a4d8aa36d9cec8, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:54.942751 478 consensus_queue.cc:162] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:54.943025 478 raft_consensus.cc:314] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:55:54.943168 478 raft_consensus.cc:370] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:54.943312 478 raft_consensus.cc:2019] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:54.944738 478 raft_consensus.cc:1970] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s I0628 23:55:54.944996 478 raft_consensus.cc:386] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" member_type: VOTER } I0628 23:55:54.947167 478 leader_election.cc:248] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:54.948777 483 raft_consensus.cc:1970] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.523s I0628 23:55:54.949096 483 raft_consensus.cc:1859] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:54.951634 483 raft_consensus.cc:464] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 1 LEADER]: Becoming Leader. State: Replica: b92976556b684c5fa4a4d8aa36d9cec8, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:54.952373 483 consensus_queue.cc:145] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" member_type: VOTER } I0628 23:55:54.971581 478 sys_catalog.cc:235] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" config { opid_index: -1 peers { permanent_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" member_type: VOTER } } I0628 23:55:54.972296 478 sys_catalog.cc:238] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [sys.catalog]: This master's current role is: LEADER I0628 23:55:54.971581 484 sys_catalog.cc:235] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92976556b684c5fa4a4d8aa36d9cec8. Latest consensus state: current_term: 1 leader_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" config { opid_index: -1 peers { permanent_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" member_type: VOTER } } I0628 23:55:54.974097 484 sys_catalog.cc:238] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [sys.catalog]: This master's current role is: LEADER I0628 23:55:54.971935 485 sys_catalog.cc:235] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" config { opid_index: -1 peers { permanent_uuid: "b92976556b684c5fa4a4d8aa36d9cec8" member_type: VOTER } } I0628 23:55:54.974774 485 sys_catalog.cc:238] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [sys.catalog]: This master's current role is: LEADER I0628 23:55:54.976511 486 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:55:54.976964 478 sys_catalog.cc:311] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:55:54.978844 487 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:55:54.988106 435 master_main.cc:64] Master server successfully started. I0628 23:55:55.037576 479 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:55.038154 479 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:55.052552 479 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:55.055547 479 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 163851 I0628 23:55:55.061668 479 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:55:55.061871 479 server_base.cc:165] Creating new FS layout I0628 23:55:55.070430 479 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "f6c5579751a648a58f62e80b545151e9" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:55.078816 479 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0 uuid: "f6c5579751a648a58f62e80b545151e9" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:55.115059 479 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:55.243396 479 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:46860 I0628 23:55:55.244819 479 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:55:55.244930 479 webserver.cc:133] Document root disabled I0628 23:55:55.246539 479 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:41921/ I0628 23:55:55.247988 479 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:55.250813 479 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:55.257344 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 479 I0628 23:55:55.258478 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43899 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:55.269031 603 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43899 I0628 23:55:55.269502 603 heartbeater.cc:319] Registering TS with master... I0628 23:55:55.269794 603 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:55.272199 455 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "f6c5579751a648a58f62e80b545151e9" instance_seqno: 1467158155218789 } with Master I0628 23:55:55.379441 607 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:55.379937 607 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:55.394094 607 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:55.397603 607 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 163851 I0628 23:55:55.404279 607 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:55:55.404480 607 server_base.cc:165] Creating new FS layout I0628 23:55:55.412633 607 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "7ca718c0188f423f8a8e39b61ae18123" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:55.421751 607 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1 uuid: "7ca718c0188f423f8a8e39b61ae18123" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:55.458400 607 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:55.566426 607 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:47459 I0628 23:55:55.568022 607 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:55:55.568184 607 webserver.cc:133] Document root disabled I0628 23:55:55.569844 607 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:36123/ I0628 23:55:55.571209 607 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:55:55.573737 607 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:55.578935 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 607 I0628 23:55:55.580006 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43899 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:55.591135 721 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43899 I0628 23:55:55.591765 721 heartbeater.cc:319] Registering TS with master... I0628 23:55:55.592072 721 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:55.593813 455 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "7ca718c0188f423f8a8e39b61ae18123" instance_seqno: 1467158155545056 } with Master I0628 23:55:55.698256 724 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:55.698736 724 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:55.712266 724 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:55.715211 724 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 163851 I0628 23:55:55.721206 724 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:55:55.721408 724 server_base.cc:165] Creating new FS layout I0628 23:55:55.729636 724 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "753f8b6ad944415c95e7fbaa4c9e3329" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:55.737206 724 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2 uuid: "753f8b6ad944415c95e7fbaa4c9e3329" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:55.769903 724 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:55.863369 724 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:49767 I0628 23:55:55.864962 724 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:55:55.865051 724 webserver.cc:133] Document root disabled I0628 23:55:55.866555 724 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:56126/ I0628 23:55:55.868010 724 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:55:55.870829 724 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:55.870925 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 724 I0628 23:55:55.892680 838 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43899 I0628 23:55:55.893173 838 heartbeater.cc:319] Registering TS with master... I0628 23:55:55.893481 838 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:55.895174 455 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "753f8b6ad944415c95e7fbaa4c9e3329" instance_seqno: 1467158155843637 } with Master I0628 23:55:55.896922 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:55:55.921118 27213 delete_table-test.cc:893] Running with fault flag: fault_crash_after_blocks_deleted W0628 23:55:55.928444 600 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:49113 forcing change of non-runtime-safe flag fault_crash_after_blocks_deleted I0628 23:55:55.928761 600 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:49113 changed flags via RPC: fault_crash_after_blocks_deleted from '0' to '1.0' W0628 23:55:55.935493 718 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:40612 forcing change of non-runtime-safe flag fault_crash_after_blocks_deleted I0628 23:55:55.935753 718 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:40612 changed flags via RPC: fault_crash_after_blocks_deleted from '0' to '1.0' W0628 23:55:55.942526 835 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:50468 forcing change of non-runtime-safe flag fault_crash_after_blocks_deleted I0628 23:55:55.942790 835 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:50468 changed flags via RPC: fault_crash_after_blocks_deleted from '0' to '1.0' I0628 23:55:55.957931 454 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:50102: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:55:56.003885 540 tablet_service.cc:612] Processing CreateTablet for tablet 9159f37006d34af4bc25489d3d6a5c56 (table=test-workload [id=04ac5d5073fe4c56beaf6d5e2e49d0dd]), partition=range: [(), ()) I0628 23:55:56.005192 658 tablet_service.cc:612] Processing CreateTablet for tablet 9159f37006d34af4bc25489d3d6a5c56 (table=test-workload [id=04ac5d5073fe4c56beaf6d5e2e49d0dd]), partition=range: [(), ()) I0628 23:55:56.009230 775 tablet_service.cc:612] Processing CreateTablet for tablet 9159f37006d34af4bc25489d3d6a5c56 (table=test-workload [id=04ac5d5073fe4c56beaf6d5e2e49d0dd]), partition=range: [(), ()) I0628 23:55:56.022913 658 ts_tablet_manager.cc:760] Registered tablet 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:56.026988 540 ts_tablet_manager.cc:760] Registered tablet 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:56.027490 858 ts_tablet_manager.cc:628] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Bootstrapping tablet I0628 23:55:56.031126 858 tablet_bootstrap.cc:393] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Bootstrap starting. I0628 23:55:56.033200 775 ts_tablet_manager.cc:760] Registered tablet 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:56.036201 858 tablet_bootstrap.cc:553] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:55:56.036825 858 tablet_bootstrap.cc:493] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: No blocks or log segments found. Creating new log. I0628 23:55:56.039044 858 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:56.040865 860 ts_tablet_manager.cc:628] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Bootstrapping tablet I0628 23:55:56.044056 860 tablet_bootstrap.cc:393] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Bootstrap starting. I0628 23:55:56.049203 860 tablet_bootstrap.cc:553] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Time spent opening tablet: real 0.002s user 0.004s sys 0.000s I0628 23:55:56.049263 861 ts_tablet_manager.cc:628] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Bootstrapping tablet I0628 23:55:56.049804 860 tablet_bootstrap.cc:493] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: No blocks or log segments found. Creating new log. I0628 23:55:56.051863 860 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:56.052383 861 tablet_bootstrap.cc:393] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Bootstrap starting. I0628 23:55:56.055575 858 tablet_bootstrap.cc:393] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: No bootstrap required, opened a new log I0628 23:55:56.056069 858 ts_tablet_manager.cc:633] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.008s I0628 23:55:56.056855 860 tablet_bootstrap.cc:393] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: No bootstrap required, opened a new log I0628 23:55:56.057230 860 ts_tablet_manager.cc:633] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s I0628 23:55:56.057613 861 tablet_bootstrap.cc:553] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:56.058300 861 tablet_bootstrap.cc:493] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: No blocks or log segments found. Creating new log. I0628 23:55:56.061648 861 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:56.064045 860 raft_consensus.cc:269] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f6c5579751a648a58f62e80b545151e9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46860 } } peers { permanent_uuid: "7ca718c0188f423f8a8e39b61ae18123" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 47459 } } peers { permanent_uuid: "753f8b6ad944415c95e7fbaa4c9e3329" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 49767 } } I0628 23:55:56.065994 860 raft_consensus.cc:301] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:56.066267 860 raft_consensus.cc:499] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c5579751a648a58f62e80b545151e9, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:56.065512 858 raft_consensus.cc:269] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f6c5579751a648a58f62e80b545151e9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46860 } } peers { permanent_uuid: "7ca718c0188f423f8a8e39b61ae18123" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 47459 } } peers { permanent_uuid: "753f8b6ad944415c95e7fbaa4c9e3329" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 49767 } } I0628 23:55:56.066685 860 consensus_queue.cc:162] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:56.069684 858 raft_consensus.cc:301] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:56.070006 858 raft_consensus.cc:499] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca718c0188f423f8a8e39b61ae18123, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:56.070652 858 consensus_queue.cc:162] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:56.076767 858 ts_tablet_manager.cc:659] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Time spent starting tablet: real 0.020s user 0.012s sys 0.004s I0628 23:55:56.080010 860 ts_tablet_manager.cc:659] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Time spent starting tablet: real 0.023s user 0.004s sys 0.008s I0628 23:55:56.085656 861 tablet_bootstrap.cc:393] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: No bootstrap required, opened a new log I0628 23:55:56.086038 861 ts_tablet_manager.cc:633] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.016s I0628 23:55:56.091858 861 raft_consensus.cc:269] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f6c5579751a648a58f62e80b545151e9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46860 } } peers { permanent_uuid: "7ca718c0188f423f8a8e39b61ae18123" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 47459 } } peers { permanent_uuid: "753f8b6ad944415c95e7fbaa4c9e3329" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 49767 } } I0628 23:55:56.093622 861 raft_consensus.cc:301] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:56.093891 861 raft_consensus.cc:499] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753f8b6ad944415c95e7fbaa4c9e3329, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:56.094323 861 consensus_queue.cc:162] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:56.096848 861 ts_tablet_manager.cc:659] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s I0628 23:55:56.543115 872 raft_consensus.cc:370] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:56.543354 872 raft_consensus.cc:2019] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:56.545501 872 raft_consensus.cc:1970] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.520s I0628 23:55:56.545727 872 raft_consensus.cc:386] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "f6c5579751a648a58f62e80b545151e9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46860 } } peers { permanent_uuid: "7ca718c0188f423f8a8e39b61ae18123" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 47459 } } peers { permanent_uuid: "753f8b6ad944415c95e7fbaa4c9e3329" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 49767 } } I0628 23:55:56.549000 872 leader_election.cc:223] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [CANDIDATE]: Term 1 election: Requesting vote from peer f6c5579751a648a58f62e80b545151e9 I0628 23:55:56.549641 872 leader_election.cc:223] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [CANDIDATE]: Term 1 election: Requesting vote from peer 7ca718c0188f423f8a8e39b61ae18123 I0628 23:55:56.557799 678 raft_consensus.cc:2019] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:56.557940 560 raft_consensus.cc:2019] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:56.559947 678 raft_consensus.cc:1970] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.536s I0628 23:55:56.560031 560 raft_consensus.cc:1970] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.581s I0628 23:55:56.561755 678 raft_consensus.cc:1677] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753f8b6ad944415c95e7fbaa4c9e3329 in term 1. I0628 23:55:56.562034 560 raft_consensus.cc:1677] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753f8b6ad944415c95e7fbaa4c9e3329 in term 1. I0628 23:55:56.562790 730 leader_election.cc:347] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [CANDIDATE]: Term 1 election: Vote granted by peer 7ca718c0188f423f8a8e39b61ae18123 I0628 23:55:56.563050 730 leader_election.cc:248] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:56.563544 730 leader_election.cc:347] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [CANDIDATE]: Term 1 election: Vote granted by peer f6c5579751a648a58f62e80b545151e9 I0628 23:55:56.563572 873 raft_consensus.cc:1970] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.592s I0628 23:55:56.563840 873 raft_consensus.cc:1859] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:56.565589 873 raft_consensus.cc:464] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 LEADER]: Becoming Leader. State: Replica: 753f8b6ad944415c95e7fbaa4c9e3329, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:56.566360 873 consensus_queue.cc:145] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "f6c5579751a648a58f62e80b545151e9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46860 } } peers { permanent_uuid: "7ca718c0188f423f8a8e39b61ae18123" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 47459 } } peers { permanent_uuid: "753f8b6ad944415c95e7fbaa4c9e3329" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 49767 } } I0628 23:55:56.591943 455 catalog_manager.cc:1673] Tablet: 9159f37006d34af4bc25489d3d6a5c56 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "753f8b6ad944415c95e7fbaa4c9e3329" config { opid_index: -1 peers { permanent_uuid: "f6c5579751a648a58f62e80b545151e9" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 46860 } } peers { permanent_uuid: "7ca718c0188f423f8a8e39b61ae18123" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 47459 } } peers { permanent_uuid: "753f8b6ad944415c95e7fbaa4c9e3329" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 49767 } } } W0628 23:55:56.594530 881 raft_consensus_state.cc:541] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:55:56.715561 455 catalog_manager.cc:1036] Servicing DeleteTable request from {real_user=slave, eff_user=} at 127.0.0.1:50098: table { table_name: "test-workload" } I0628 23:55:56.721866 455 catalog_manager.cc:2531] Sending DeleteTablet for 3 replicas of tablet 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:56.722000 455 catalog_manager.cc:2547] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8: Deleting tablet 9159f37006d34af4bc25489d3d6a5c56 on peer f6c5579751a648a58f62e80b545151e9 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:55:56 UTC) I0628 23:55:56.722443 455 catalog_manager.cc:2547] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8: Deleting tablet 9159f37006d34af4bc25489d3d6a5c56 on peer 7ca718c0188f423f8a8e39b61ae18123 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:55:56 UTC) I0628 23:55:56.722795 455 catalog_manager.cc:2547] T 00000000000000000000000000000000 P b92976556b684c5fa4a4d8aa36d9cec8: Deleting tablet 9159f37006d34af4bc25489d3d6a5c56 on peer 753f8b6ad944415c95e7fbaa4c9e3329 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:55:56 UTC) I0628 23:55:56.723790 540 tablet_service.cc:653] Processing DeleteTablet for tablet 9159f37006d34af4bc25489d3d6a5c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:55:56 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:49117 I0628 23:55:56.723875 658 tablet_service.cc:653] Processing DeleteTablet for tablet 9159f37006d34af4bc25489d3d6a5c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:55:56 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:40616 I0628 23:55:56.724239 540 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:56.724241 658 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:56.724301 455 catalog_manager.cc:1119] Successfully deleted table test-workload [id=04ac5d5073fe4c56beaf6d5e2e49d0dd] per request from {real_user=slave, eff_user=} at 127.0.0.1:50098 I0628 23:55:56.724669 658 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.724550 775 tablet_service.cc:653] Processing DeleteTablet for tablet 9159f37006d34af4bc25489d3d6a5c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:55:56 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:50472 I0628 23:55:56.724670 540 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.724997 540 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725008 775 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:56.725008 658 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725280 540 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725282 658 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725467 775 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725531 540 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725536 658 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725708 540 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725713 658 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725713 775 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725872 540 maintenance_manager.cc:180] Unregistered op LogGCOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.725920 775 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.726008 658 maintenance_manager.cc:180] Unregistered op LogGCOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.726066 540 raft_consensus.cc:1495] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:55:56.726152 775 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.726209 658 raft_consensus.cc:1495] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:55:56.726330 775 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.726357 540 raft_consensus.cc:1511] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:55:56.726505 775 maintenance_manager.cc:180] Unregistered op LogGCOp(9159f37006d34af4bc25489d3d6a5c56) I0628 23:55:56.726601 658 raft_consensus.cc:1511] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:55:56.726739 775 raft_consensus.cc:1495] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 LEADER]: Raft consensus shutting down. I0628 23:55:56.727710 775 consensus_peers.cc:351] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 -> Peer 7ca718c0188f423f8a8e39b61ae18123 (127.106.77.1:47459): Closing peer: 7ca718c0188f423f8a8e39b61ae18123 I0628 23:55:56.728520 775 consensus_peers.cc:351] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 -> Peer f6c5579751a648a58f62e80b545151e9 (127.106.77.0:46860): Closing peer: f6c5579751a648a58f62e80b545151e9 I0628 23:55:56.729429 775 raft_consensus_state.cc:377] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 LEADER]: Trying to abort 1 pending transactions. I0628 23:55:56.729665 775 raft_consensus_state.cc:382] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 LEADER]: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6009479809313484800 op_type: NO_OP noop_request { } I0628 23:55:56.730303 775 raft_consensus.cc:1906] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: NO_OP replication failed: Aborted: Transaction aborted I0628 23:55:56.730576 775 raft_consensus.cc:1511] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329 [term 1 LEADER]: Raft consensus is shut down! I0628 23:55:56.731024 540 ts_tablet_manager.cc:955] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:55:56.732686 658 ts_tablet_manager.cc:955] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:55:56.735193 540 ts_tablet_manager.cc:965] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Tablet deleted. Last logged OpId: 0.0 W0628 23:55:56.735371 540 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:55:56.735453 540 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:55:56.735587 540 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_blocks_deleted *** Check failure stack trace: *** I0628 23:55:56.736640 775 ts_tablet_manager.cc:955] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Deleting tablet data with delete state TABLET_DATA_DELETED @ 0x7fb064cd9304 google::LogMessage::SendToLog() at ??:0 I0628 23:55:56.739212 658 ts_tablet_manager.cc:965] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Tablet deleted. Last logged OpId: 0.0 W0628 23:55:56.739457 658 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:55:56.739578 658 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:55:56.739753 658 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_blocks_deleted *** Check failure stack trace: *** I0628 23:55:56.743140 775 ts_tablet_manager.cc:965] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Tablet deleted. Last logged OpId: 1.1 W0628 23:55:56.743571 775 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:55:56.743765 775 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:55:56.743922 775 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_blocks_deleted *** Check failure stack trace: *** @ 0x7fb064cd9800 google::LogMessage::Flush() at ??:0 @ 0x7fa316b85304 google::LogMessage::SendToLog() at ??:0 @ 0x7fb064cdd432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7fa576763304 google::LogMessage::SendToLog() at ??:0 @ 0x7fa316b85800 google::LogMessage::Flush() at ??:0 @ 0x7fb0659a4e42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7fb06c860e6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7fa316b89432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7fa576763800 google::LogMessage::Flush() at ??:0 @ 0x7fb06c8a83c4 kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7fa317850e42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7fa576767432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7fb06c8a91dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7fa31e70ce6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7fb06c886497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7fa57742ee42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7fa31e7543c4 kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7fb06a444ef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7fb06a444d0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7fa31e7551dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7fb069f75d49 std::function<>::operator()() at ??:0 @ 0x7fa31e732497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7fa57e2eae6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7fb069f75949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7fa31c2f0ef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7fb069f7769e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7fa57e3323c4 kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7fa31c2f0d0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7fb069f79477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7fa57e3331dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7fa31be21d49 std::function<>::operator()() at ??:0 @ 0x7fb069f793db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7fa31be21949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7fa57e310497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7fb069f79384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7fa31be2369e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7fa57beceef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7fb069f791aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7fa31be25477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7fb069f20e72 boost::function0<>::operator()() at ??:0 @ 0x7fa57beced0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7fa31be253db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7fb065a0f560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7fa57b9ffd49 std::function<>::operator()() at ??:0 @ 0x7fa31be25384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7fb06744e182 start_thread at ??:0 @ 0x7fa31be251aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7fa57b9ff949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7fb062fdd47d clone at ??:0 @ 0x7fa31bdcce72 boost::function0<>::operator()() at ??:0 @ 0x7fa57ba0169e kudu::rpc::ServicePool::RunThread() at ??:0 @ (nil) (unknown) @ 0x7fa3178bb560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7fa57ba03477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7fa3192fa182 start_thread at ??:0 @ 0x7fa314e8947d clone at ??:0 @ 0x7fa57ba033db boost::_bi::list1<>::operator()<>() at ??:0 @ (nil) (unknown) @ 0x7fa57ba03384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7fa57ba031aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7fa57b9aae72 boost::function0<>::operator()() at ??:0 @ 0x7fa577499560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 W0628 23:55:56.877200 896 catalog_manager.cc:1957] TS 7ca718c0188f423f8a8e39b61ae18123: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Recv() got EOF from remote (error 108) I0628 23:55:56.877511 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=7ca718c0188f423f8a8e39b61ae18123 with a delay of 60ms (attempt = 1)... W0628 23:55:56.878303 896 catalog_manager.cc:1957] TS f6c5579751a648a58f62e80b545151e9: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Recv() got EOF from remote (error 108) I0628 23:55:56.878443 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=f6c5579751a648a58f62e80b545151e9 with a delay of 54ms (attempt = 1)... @ 0x7fa578ed8182 start_thread at ??:0 @ 0x7fa574a6747d clone at ??:0 @ (nil) (unknown) W0628 23:55:56.905864 896 catalog_manager.cc:1957] TS 753f8b6ad944415c95e7fbaa4c9e3329: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Recv() got EOF from remote (error 108) I0628 23:55:56.906005 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=753f8b6ad944415c95e7fbaa4c9e3329 with a delay of 42ms (attempt = 1)... I0628 23:55:56.919077 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:46860 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=41921 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:43899 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 W0628 23:55:56.936550 896 catalog_manager.cc:1957] TS f6c5579751a648a58f62e80b545151e9: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.0:46860: connect: Connection refused (error 111) I0628 23:55:56.936779 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=f6c5579751a648a58f62e80b545151e9 with a delay of 51ms (attempt = 2)... W0628 23:55:56.939249 896 catalog_manager.cc:1957] TS 7ca718c0188f423f8a8e39b61ae18123: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.1:47459: connect: Connection refused (error 111) I0628 23:55:56.939416 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=7ca718c0188f423f8a8e39b61ae18123 with a delay of 75ms (attempt = 2)... W0628 23:55:56.950676 896 catalog_manager.cc:1957] TS 753f8b6ad944415c95e7fbaa4c9e3329: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.2:49767: connect: Connection refused (error 111) I0628 23:55:56.950886 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=753f8b6ad944415c95e7fbaa4c9e3329 with a delay of 38ms (attempt = 2)... W0628 23:55:56.990201 896 catalog_manager.cc:1957] TS f6c5579751a648a58f62e80b545151e9: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.0:46860: connect: Connection refused (error 111) I0628 23:55:56.990437 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=f6c5579751a648a58f62e80b545151e9 with a delay of 89ms (attempt = 3)... W0628 23:55:56.991658 896 catalog_manager.cc:1957] TS 753f8b6ad944415c95e7fbaa4c9e3329: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.2:49767: connect: Connection refused (error 111) I0628 23:55:56.991894 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=753f8b6ad944415c95e7fbaa4c9e3329 with a delay of 75ms (attempt = 3)... W0628 23:55:57.016361 896 catalog_manager.cc:1957] TS 7ca718c0188f423f8a8e39b61ae18123: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.1:47459: connect: Connection refused (error 111) I0628 23:55:57.016516 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=7ca718c0188f423f8a8e39b61ae18123 with a delay of 76ms (attempt = 3)... I0628 23:55:57.046167 906 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:57.046625 906 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:57.066557 906 tablet_server_main.cc:54] Initializing tablet server... W0628 23:55:57.069463 896 catalog_manager.cc:1957] TS 753f8b6ad944415c95e7fbaa4c9e3329: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.2:49767: connect: Connection refused (error 111) I0628 23:55:57.069614 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=753f8b6ad944415c95e7fbaa4c9e3329 with a delay of 157ms (attempt = 4)... I0628 23:55:57.071075 906 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 164851 W0628 23:55:57.081465 896 catalog_manager.cc:1957] TS f6c5579751a648a58f62e80b545151e9: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.0:46860: connect: Connection refused (error 111) I0628 23:55:57.081609 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=f6c5579751a648a58f62e80b545151e9 with a delay of 146ms (attempt = 4)... I0628 23:55:57.084419 906 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0 uuid: "f6c5579751a648a58f62e80b545151e9" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" W0628 23:55:57.094753 896 catalog_manager.cc:1957] TS 7ca718c0188f423f8a8e39b61ae18123: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.1:47459: connect: Connection refused (error 111) I0628 23:55:57.094905 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=7ca718c0188f423f8a8e39b61ae18123 with a delay of 130ms (attempt = 4)... I0628 23:55:57.121111 906 ts_tablet_manager.cc:604] Loading metadata for tablet 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:57.126215 906 ts_tablet_manager.cc:936] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:55:57.132489 906 ts_tablet_manager.cc:955] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:55:57.138290 906 ts_tablet_manager.cc:965] T 9159f37006d34af4bc25489d3d6a5c56 P f6c5579751a648a58f62e80b545151e9: Tablet deleted. Last logged OpId: 0.0 I0628 23:55:57.140202 906 tablet_server_main.cc:57] Starting tablet server... W0628 23:55:57.227967 896 catalog_manager.cc:1957] TS 7ca718c0188f423f8a8e39b61ae18123: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.1:47459: connect: Connection refused (error 111) I0628 23:55:57.228212 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=7ca718c0188f423f8a8e39b61ae18123 with a delay of 263ms (attempt = 5)... W0628 23:55:57.229208 896 catalog_manager.cc:1957] TS 753f8b6ad944415c95e7fbaa4c9e3329: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.2:49767: connect: Connection refused (error 111) I0628 23:55:57.229398 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=753f8b6ad944415c95e7fbaa4c9e3329 with a delay of 272ms (attempt = 5)... W0628 23:55:57.229980 896 catalog_manager.cc:1957] TS f6c5579751a648a58f62e80b545151e9: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.0:46860: connect: Connection refused (error 111) I0628 23:55:57.230202 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=f6c5579751a648a58f62e80b545151e9 with a delay of 283ms (attempt = 5)... I0628 23:55:57.233052 906 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:46860 I0628 23:55:57.234460 906 webserver.cc:122] Starting webserver on 127.106.77.0:41921 I0628 23:55:57.234560 906 webserver.cc:133] Document root disabled I0628 23:55:57.236071 906 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:41921/ I0628 23:55:57.237471 906 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:57.238467 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 906 I0628 23:55:57.239184 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:47459 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_port=36123 --webserver_interface=127.106.77.1 --tserver_master_addrs=127.0.0.1:43899 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:57.240249 906 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:57.249917 1061 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43899 I0628 23:55:57.250222 1061 heartbeater.cc:319] Registering TS with master... I0628 23:55:57.250602 1061 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:57.252216 455 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "f6c5579751a648a58f62e80b545151e9" instance_seqno: 1467158157213036 } with Master I0628 23:55:57.353308 1064 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:57.353737 1064 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:57.367097 1064 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:57.369976 1064 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 164851 I0628 23:55:57.381781 1064 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1 uuid: "7ca718c0188f423f8a8e39b61ae18123" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:57.415879 1064 ts_tablet_manager.cc:604] Loading metadata for tablet 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:57.421098 1064 ts_tablet_manager.cc:936] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:55:57.427672 1064 ts_tablet_manager.cc:955] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:55:57.433254 1064 ts_tablet_manager.cc:965] T 9159f37006d34af4bc25489d3d6a5c56 P 7ca718c0188f423f8a8e39b61ae18123: Tablet deleted. Last logged OpId: 0.0 I0628 23:55:57.435307 1064 tablet_server_main.cc:57] Starting tablet server... W0628 23:55:57.494961 896 catalog_manager.cc:1957] TS 7ca718c0188f423f8a8e39b61ae18123: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.1:47459: connect: Connection refused (error 111) I0628 23:55:57.495249 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=7ca718c0188f423f8a8e39b61ae18123 with a delay of 534ms (attempt = 6)... W0628 23:55:57.503759 896 catalog_manager.cc:1957] TS 753f8b6ad944415c95e7fbaa4c9e3329: Delete Tablet RPC failed for tablet 9159f37006d34af4bc25489d3d6a5c56: Network error: Client connection negotiation failed: client connection to 127.106.77.2:49767: connect: Connection refused (error 111) I0628 23:55:57.503903 896 catalog_manager.cc:2014] Scheduling retry of 9159f37006d34af4bc25489d3d6a5c56 Delete Tablet RPC for TS=753f8b6ad944415c95e7fbaa4c9e3329 with a delay of 519ms (attempt = 6)... I0628 23:55:57.521643 998 tablet_service.cc:653] Processing DeleteTablet for tablet 9159f37006d34af4bc25489d3d6a5c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:55:56 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:49134 W0628 23:55:57.523787 896 catalog_manager.cc:2207] TS f6c5579751a648a58f62e80b545151e9: delete failed for tablet 9159f37006d34af4bc25489d3d6a5c56 because the tablet was not found. No further retry: Not found: Tablet not found: 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:57.527354 1064 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:47459 I0628 23:55:57.528728 1064 webserver.cc:122] Starting webserver on 127.106.77.1:36123 I0628 23:55:57.528831 1064 webserver.cc:133] Document root disabled I0628 23:55:57.530124 1064 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:36123/ I0628 23:55:57.531282 1064 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:55:57.534205 1064 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:57.538455 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 1064 I0628 23:55:57.539552 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:49767 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_port=56126 --webserver_interface=127.106.77.2 --tserver_master_addrs=127.0.0.1:43899 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:57.543493 1178 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43899 I0628 23:55:57.543927 1178 heartbeater.cc:319] Registering TS with master... I0628 23:55:57.544211 1178 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:57.545910 455 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "7ca718c0188f423f8a8e39b61ae18123" instance_seqno: 1467158157507228 } with Master I0628 23:55:57.651253 1181 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:57.651864 1181 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:57.665199 1181 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:57.668000 1181 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 164851 I0628 23:55:57.678892 1181 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2 uuid: "753f8b6ad944415c95e7fbaa4c9e3329" format_stamp: "Formatted at 2016-06-28 23:55:55 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:57.712373 1181 ts_tablet_manager.cc:604] Loading metadata for tablet 9159f37006d34af4bc25489d3d6a5c56 I0628 23:55:57.719458 1181 ts_tablet_manager.cc:936] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:55:57.725276 1181 ts_tablet_manager.cc:955] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:55:57.731134 1181 ts_tablet_manager.cc:965] T 9159f37006d34af4bc25489d3d6a5c56 P 753f8b6ad944415c95e7fbaa4c9e3329: Tablet deleted. Last logged OpId: 1.1 I0628 23:55:57.733117 1181 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:57.823925 1181 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:49767 I0628 23:55:57.825350 1181 webserver.cc:122] Starting webserver on 127.106.77.2:56126 I0628 23:55:57.825479 1181 webserver.cc:133] Document root disabled I0628 23:55:57.827006 1181 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:56126/ I0628 23:55:57.828562 1181 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_0.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:55:57.830967 1181 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:57.838557 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 1181 I0628 23:55:57.840380 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 435 I0628 23:55:57.840399 1295 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:43899 I0628 23:55:57.840795 1295 heartbeater.cc:319] Registering TS with master... I0628 23:55:57.841065 1295 heartbeater.cc:325] Sending a full tablet report to master... W0628 23:55:57.857604 1187 connection.cc:458] client connection to 127.0.0.1:43899 recv error: Network error: recv error: Connection reset by peer (error 104) W0628 23:55:57.858258 1295 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:43899: Network error: Failed to send heartbeat: recv error: Connection reset by peer (error 104) I0628 23:55:57.858582 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 906 I0628 23:55:57.858641 1295 heartbeater.cc:319] Registering TS with master... I0628 23:55:57.858816 1295 heartbeater.cc:325] Sending a full tablet report to master... W0628 23:55:57.861003 1295 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:43899: Network error: Failed to send heartbeat: Client connection negotiation failed: client connection to 127.0.0.1:43899: connect: Connection refused (error 111) I0628 23:55:57.861280 1295 heartbeater.cc:319] Registering TS with master... I0628 23:55:57.861441 1295 heartbeater.cc:325] Sending a full tablet report to master... W0628 23:55:57.863070 1295 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:43899: Network error: Failed to send heartbeat: Client connection negotiation failed: client connection to 127.0.0.1:43899: connect: Connection refused (error 111) W0628 23:55:57.863227 1295 heartbeater.cc:283] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0628 23:55:57.863430 1295 heartbeater.cc:319] Registering TS with master... I0628 23:55:57.863580 1295 heartbeater.cc:325] Sending a full tablet report to master... W0628 23:55:57.864874 1295 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:43899: Network error: Failed to send heartbeat: Client connection negotiation failed: client connection to 127.0.0.1:43899: connect: Connection refused (error 111) I0628 23:55:57.874136 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 1064 I0628 23:55:57.887420 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 1181 [ OK ] FaultFlags/DeleteTableDeletedParamTest.TestRollForwardDelete/0 (3251 ms) [ RUN ] FaultFlags/DeleteTableDeletedParamTest.TestRollForwardDelete/1 I0628 23:55:57.909976 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:58.027173 1299 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:58.027698 1299 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:58.041070 1299 master_main.cc:58] Initializing master server... I0628 23:55:58.043545 1299 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 165351 I0628 23:55:58.050102 1299 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:55:58.050314 1299 server_base.cc:165] Creating new FS layout I0628 23:55:58.063346 1299 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/master/instance: uuid: "d1a7d15d9c4a4699a52a980ba690d0df" format_stamp: "Formatted at 2016-06-28 23:55:58 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:58.076282 1299 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/master uuid: "d1a7d15d9c4a4699a52a980ba690d0df" format_stamp: "Formatted at 2016-06-28 23:55:58 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:58.109359 1299 master_main.cc:61] Starting Master server... I0628 23:55:58.145784 1299 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:55050 I0628 23:55:58.147212 1299 webserver.cc:122] Starting webserver on localhost:0 I0628 23:55:58.147303 1299 webserver.cc:133] Document root disabled I0628 23:55:58.148052 1299 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:55:58.149341 1299 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:57105/ I0628 23:55:58.150743 1299 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:55:58.157449 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 1299 I0628 23:55:58.158223 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:55050 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:58.176928 1342 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df: Bootstrap starting. I0628 23:55:58.180641 1342 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df: Time spent opening tablet: real 0.001s user 0.004s sys 0.000s I0628 23:55:58.181159 1342 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df: No blocks or log segments found. Creating new log. I0628 23:55:58.182709 1342 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:58.188432 1342 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df: No bootstrap required, opened a new log I0628 23:55:58.193825 1342 raft_consensus.cc:269] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" member_type: VOTER } I0628 23:55:58.194715 1342 raft_consensus.cc:301] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:58.194929 1342 raft_consensus.cc:499] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a7d15d9c4a4699a52a980ba690d0df, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:58.195356 1342 consensus_queue.cc:162] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:58.195688 1342 raft_consensus.cc:314] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:55:58.195828 1342 raft_consensus.cc:370] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:58.195945 1342 raft_consensus.cc:2019] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:58.197779 1342 raft_consensus.cc:1970] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.511s I0628 23:55:58.198076 1342 raft_consensus.cc:386] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" member_type: VOTER } I0628 23:55:58.200714 1342 leader_election.cc:248] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:58.202375 1347 raft_consensus.cc:1970] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.507s I0628 23:55:58.202746 1347 raft_consensus.cc:1859] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:58.205070 1347 raft_consensus.cc:464] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 1 LEADER]: Becoming Leader. State: Replica: d1a7d15d9c4a4699a52a980ba690d0df, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:58.205847 1347 consensus_queue.cc:145] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" member_type: VOTER } I0628 23:55:58.226214 1342 sys_catalog.cc:235] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" config { opid_index: -1 peers { permanent_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" member_type: VOTER } } I0628 23:55:58.226384 1348 sys_catalog.cc:235] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" config { opid_index: -1 peers { permanent_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" member_type: VOTER } } I0628 23:55:58.227283 1348 sys_catalog.cc:238] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [sys.catalog]: This master's current role is: LEADER I0628 23:55:58.227267 1342 sys_catalog.cc:238] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [sys.catalog]: This master's current role is: LEADER I0628 23:55:58.226860 1349 sys_catalog.cc:235] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a7d15d9c4a4699a52a980ba690d0df. Latest consensus state: current_term: 1 leader_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" config { opid_index: -1 peers { permanent_uuid: "d1a7d15d9c4a4699a52a980ba690d0df" member_type: VOTER } } I0628 23:55:58.233180 1349 sys_catalog.cc:238] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [sys.catalog]: This master's current role is: LEADER I0628 23:55:58.234308 1350 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:55:58.235157 1342 sys_catalog.cc:311] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [sys.catalog]: configured and running, proceeding with master startup. I0628 23:55:58.237488 1351 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:55:58.252924 1299 master_main.cc:64] Master server successfully started. I0628 23:55:58.284075 1343 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:58.284592 1343 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:58.308989 1343 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:58.312059 1343 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 165351 I0628 23:55:58.318198 1343 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:55:58.318424 1343 server_base.cc:165] Creating new FS layout I0628 23:55:58.327246 1343 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "35c51db271ff443f872fc364af318f22" format_stamp: "Formatted at 2016-06-28 23:55:58 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:58.335841 1343 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0 uuid: "35c51db271ff443f872fc364af318f22" format_stamp: "Formatted at 2016-06-28 23:55:58 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:58.392623 1343 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:58.567834 1343 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:34645 I0628 23:55:58.569427 1343 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:55:58.569542 1343 webserver.cc:133] Document root disabled I0628 23:55:58.571140 1343 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:42992/ I0628 23:55:58.572572 1343 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:55:58.578191 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 1343 I0628 23:55:58.579222 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:55050 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:58.588614 1343 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:58.600630 1467 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:55050 I0628 23:55:58.601130 1467 heartbeater.cc:319] Registering TS with master... I0628 23:55:58.601500 1467 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:58.614099 1319 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "35c51db271ff443f872fc364af318f22" instance_seqno: 1467158158525780 } with Master I0628 23:55:58.711333 1471 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:58.711854 1471 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:58.725044 1471 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:58.727674 1471 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 165351 I0628 23:55:58.733629 1471 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:55:58.733862 1471 server_base.cc:165] Creating new FS layout I0628 23:55:58.741696 1471 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "ab59d0275deb4477a827b67f8fdce34f" format_stamp: "Formatted at 2016-06-28 23:55:58 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:58.751123 1471 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1 uuid: "ab59d0275deb4477a827b67f8fdce34f" format_stamp: "Formatted at 2016-06-28 23:55:58 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:58.804576 1471 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:58.950601 1471 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:60903 I0628 23:55:58.952425 1471 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:55:58.952534 1471 webserver.cc:133] Document root disabled I0628 23:55:58.953953 1471 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:43509/ I0628 23:55:58.955889 1471 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:55:58.958169 1471 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:58.959319 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 1471 I0628 23:55:58.960314 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:55050 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:55:58.968690 1585 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:55050 I0628 23:55:58.969219 1585 heartbeater.cc:319] Registering TS with master... I0628 23:55:58.969514 1585 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:58.970885 1319 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "ab59d0275deb4477a827b67f8fdce34f" instance_seqno: 1467158158932829 } with Master I0628 23:55:59.074458 1588 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:55:59.075094 1588 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:55:59.088793 1588 tablet_server_main.cc:54] Initializing tablet server... I0628 23:55:59.091508 1588 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 165851 I0628 23:55:59.097076 1588 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:55:59.097306 1588 server_base.cc:165] Creating new FS layout I0628 23:55:59.105010 1588 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "c008181728d9485f8406ec2c378f39f7" format_stamp: "Formatted at 2016-06-28 23:55:59 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:59.113145 1588 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2 uuid: "c008181728d9485f8406ec2c378f39f7" format_stamp: "Formatted at 2016-06-28 23:55:59 on dist-test-slave-dist-test-slave-cgs5" I0628 23:55:59.153969 1588 tablet_server_main.cc:57] Starting tablet server... I0628 23:55:59.249325 1588 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:32804 I0628 23:55:59.250730 1588 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:55:59.250823 1588 webserver.cc:133] Document root disabled I0628 23:55:59.252338 1588 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:42816/ I0628 23:55:59.253621 1588 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:55:59.255733 1588 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:55:59.258926 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 1588 I0628 23:55:59.265593 1702 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:55050 I0628 23:55:59.265918 1702 heartbeater.cc:319] Registering TS with master... I0628 23:55:59.266229 1702 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:55:59.267473 1319 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "c008181728d9485f8406ec2c378f39f7" instance_seqno: 1467158159230057 } with Master I0628 23:55:59.269618 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:55:59.281538 27213 delete_table-test.cc:893] Running with fault flag: fault_crash_after_wal_deleted W0628 23:55:59.287955 1464 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:53385 forcing change of non-runtime-safe flag fault_crash_after_wal_deleted I0628 23:55:59.288311 1464 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:53385 changed flags via RPC: fault_crash_after_wal_deleted from '0' to '1.0' W0628 23:55:59.294517 1582 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:42603 forcing change of non-runtime-safe flag fault_crash_after_wal_deleted I0628 23:55:59.294806 1582 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:42603 changed flags via RPC: fault_crash_after_wal_deleted from '0' to '1.0' W0628 23:55:59.301534 1699 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:52054 forcing change of non-runtime-safe flag fault_crash_after_wal_deleted I0628 23:55:59.301801 1699 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:52054 changed flags via RPC: fault_crash_after_wal_deleted from '0' to '1.0' I0628 23:55:59.315299 1319 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:57281: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:55:59.357666 1522 tablet_service.cc:612] Processing CreateTablet for tablet a7f90e4fdbe34d30a37b6515985c2e34 (table=test-workload [id=968416e56b4e4cb8aa4f4dc337a14a4a]), partition=range: [(), ()) I0628 23:55:59.357792 1404 tablet_service.cc:612] Processing CreateTablet for tablet a7f90e4fdbe34d30a37b6515985c2e34 (table=test-workload [id=968416e56b4e4cb8aa4f4dc337a14a4a]), partition=range: [(), ()) I0628 23:55:59.363752 1639 tablet_service.cc:612] Processing CreateTablet for tablet a7f90e4fdbe34d30a37b6515985c2e34 (table=test-workload [id=968416e56b4e4cb8aa4f4dc337a14a4a]), partition=range: [(), ()) I0628 23:55:59.375349 1522 ts_tablet_manager.cc:760] Registered tablet a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:55:59.377145 1722 ts_tablet_manager.cc:628] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Bootstrapping tablet I0628 23:55:59.381822 1404 ts_tablet_manager.cc:760] Registered tablet a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:55:59.382711 1722 tablet_bootstrap.cc:393] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Bootstrap starting. I0628 23:55:59.387444 1639 ts_tablet_manager.cc:760] Registered tablet a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:55:59.390522 1722 tablet_bootstrap.cc:553] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:59.391526 1722 tablet_bootstrap.cc:493] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: No blocks or log segments found. Creating new log. I0628 23:55:59.396411 1722 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:59.400557 1725 ts_tablet_manager.cc:628] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Bootstrapping tablet I0628 23:55:59.402817 1725 tablet_bootstrap.cc:393] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Bootstrap starting. I0628 23:55:59.406708 1725 tablet_bootstrap.cc:553] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:59.407409 1725 tablet_bootstrap.cc:493] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: No blocks or log segments found. Creating new log. I0628 23:55:59.408473 1724 ts_tablet_manager.cc:628] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Bootstrapping tablet I0628 23:55:59.409231 1725 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:59.411626 1724 tablet_bootstrap.cc:393] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Bootstrap starting. I0628 23:55:59.416004 1724 tablet_bootstrap.cc:553] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:55:59.416841 1724 tablet_bootstrap.cc:493] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: No blocks or log segments found. Creating new log. I0628 23:55:59.417057 1722 tablet_bootstrap.cc:393] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: No bootstrap required, opened a new log I0628 23:55:59.417529 1722 ts_tablet_manager.cc:633] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.012s I0628 23:55:59.419641 1724 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:55:59.425945 1725 tablet_bootstrap.cc:393] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: No bootstrap required, opened a new log I0628 23:55:59.426307 1725 ts_tablet_manager.cc:633] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.012s I0628 23:55:59.432322 1722 raft_consensus.cc:269] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "ab59d0275deb4477a827b67f8fdce34f" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 60903 } } peers { permanent_uuid: "35c51db271ff443f872fc364af318f22" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34645 } } peers { permanent_uuid: "c008181728d9485f8406ec2c378f39f7" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 32804 } } I0628 23:55:59.433972 1722 raft_consensus.cc:301] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:59.433018 1725 raft_consensus.cc:269] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "ab59d0275deb4477a827b67f8fdce34f" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 60903 } } peers { permanent_uuid: "35c51db271ff443f872fc364af318f22" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34645 } } peers { permanent_uuid: "c008181728d9485f8406ec2c378f39f7" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 32804 } } I0628 23:55:59.434221 1722 raft_consensus.cc:499] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab59d0275deb4477a827b67f8fdce34f, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:59.434674 1722 consensus_queue.cc:162] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:59.434949 1725 raft_consensus.cc:301] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:59.435210 1725 raft_consensus.cc:499] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c008181728d9485f8406ec2c378f39f7, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:59.435711 1725 consensus_queue.cc:162] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:59.437755 1724 tablet_bootstrap.cc:393] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: No bootstrap required, opened a new log I0628 23:55:59.438140 1724 ts_tablet_manager.cc:633] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.004s I0628 23:55:59.439076 1725 ts_tablet_manager.cc:659] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0628 23:55:59.448667 1722 ts_tablet_manager.cc:659] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Time spent starting tablet: real 0.031s user 0.012s sys 0.004s I0628 23:55:59.460782 1724 raft_consensus.cc:269] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "ab59d0275deb4477a827b67f8fdce34f" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 60903 } } peers { permanent_uuid: "35c51db271ff443f872fc364af318f22" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34645 } } peers { permanent_uuid: "c008181728d9485f8406ec2c378f39f7" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 32804 } } I0628 23:55:59.462378 1724 raft_consensus.cc:301] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:55:59.462646 1724 raft_consensus.cc:499] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c51db271ff443f872fc364af318f22, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:59.463167 1724 consensus_queue.cc:162] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:55:59.466464 1724 ts_tablet_manager.cc:659] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Time spent starting tablet: real 0.028s user 0.008s sys 0.004s I0628 23:55:59.912134 1736 raft_consensus.cc:370] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:55:59.912325 1736 raft_consensus.cc:2019] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:59.914335 1736 raft_consensus.cc:1970] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.588s I0628 23:55:59.914635 1736 raft_consensus.cc:386] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "ab59d0275deb4477a827b67f8fdce34f" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 60903 } } peers { permanent_uuid: "35c51db271ff443f872fc364af318f22" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34645 } } peers { permanent_uuid: "c008181728d9485f8406ec2c378f39f7" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 32804 } } I0628 23:55:59.918104 1736 leader_election.cc:223] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [CANDIDATE]: Term 1 election: Requesting vote from peer ab59d0275deb4477a827b67f8fdce34f I0628 23:55:59.918589 1736 leader_election.cc:223] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [CANDIDATE]: Term 1 election: Requesting vote from peer c008181728d9485f8406ec2c378f39f7 I0628 23:55:59.926925 1542 raft_consensus.cc:2019] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:59.927098 1659 raft_consensus.cc:2019] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:55:59.929096 1542 raft_consensus.cc:1970] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.503s I0628 23:55:59.929363 1659 raft_consensus.cc:1970] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.615s I0628 23:55:59.930744 1542 raft_consensus.cc:1677] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c51db271ff443f872fc364af318f22 in term 1. I0628 23:55:59.931812 1359 leader_election.cc:347] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [CANDIDATE]: Term 1 election: Vote granted by peer ab59d0275deb4477a827b67f8fdce34f I0628 23:55:59.931859 1659 raft_consensus.cc:1677] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c51db271ff443f872fc364af318f22 in term 1. I0628 23:55:59.932050 1359 leader_election.cc:248] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:55:59.932627 1737 raft_consensus.cc:1970] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.612s I0628 23:55:59.932924 1737 raft_consensus.cc:1859] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:55:59.933015 1358 leader_election.cc:347] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [CANDIDATE]: Term 1 election: Vote granted by peer c008181728d9485f8406ec2c378f39f7 I0628 23:55:59.934327 1737 raft_consensus.cc:464] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 LEADER]: Becoming Leader. State: Replica: 35c51db271ff443f872fc364af318f22, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:55:59.935117 1737 consensus_queue.cc:145] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ab59d0275deb4477a827b67f8fdce34f" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 60903 } } peers { permanent_uuid: "35c51db271ff443f872fc364af318f22" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34645 } } peers { permanent_uuid: "c008181728d9485f8406ec2c378f39f7" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 32804 } } W0628 23:55:59.962371 1745 raft_consensus_state.cc:541] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:55:59.961539 1317 catalog_manager.cc:1673] Tablet: a7f90e4fdbe34d30a37b6515985c2e34 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "35c51db271ff443f872fc364af318f22" config { opid_index: -1 peers { permanent_uuid: "ab59d0275deb4477a827b67f8fdce34f" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 60903 } } peers { permanent_uuid: "35c51db271ff443f872fc364af318f22" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 34645 } } peers { permanent_uuid: "c008181728d9485f8406ec2c378f39f7" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 32804 } } } I0628 23:56:00.074265 1317 catalog_manager.cc:1036] Servicing DeleteTable request from {real_user=slave, eff_user=} at 127.0.0.1:57277: table { table_name: "test-workload" } I0628 23:56:00.080317 1317 catalog_manager.cc:2531] Sending DeleteTablet for 3 replicas of tablet a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:56:00.080443 1317 catalog_manager.cc:2547] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df: Deleting tablet a7f90e4fdbe34d30a37b6515985c2e34 on peer ab59d0275deb4477a827b67f8fdce34f with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:00 UTC) I0628 23:56:00.080991 1317 catalog_manager.cc:2547] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df: Deleting tablet a7f90e4fdbe34d30a37b6515985c2e34 on peer 35c51db271ff443f872fc364af318f22 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:00 UTC) I0628 23:56:00.081357 1317 catalog_manager.cc:2547] T 00000000000000000000000000000000 P d1a7d15d9c4a4699a52a980ba690d0df: Deleting tablet a7f90e4fdbe34d30a37b6515985c2e34 on peer c008181728d9485f8406ec2c378f39f7 with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:00 UTC) I0628 23:56:00.081939 1317 catalog_manager.cc:1119] Successfully deleted table test-workload [id=968416e56b4e4cb8aa4f4dc337a14a4a] per request from {real_user=slave, eff_user=} at 127.0.0.1:57277 I0628 23:56:00.082231 1522 tablet_service.cc:653] Processing DeleteTablet for tablet a7f90e4fdbe34d30a37b6515985c2e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:00 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:42606 I0628 23:56:00.082470 1404 tablet_service.cc:653] Processing DeleteTablet for tablet a7f90e4fdbe34d30a37b6515985c2e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:00 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:53390 I0628 23:56:00.082659 1522 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:56:00.082949 1404 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:56:00.083075 1639 tablet_service.cc:653] Processing DeleteTablet for tablet a7f90e4fdbe34d30a37b6515985c2e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:00 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:52058 I0628 23:56:00.083250 1522 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.083442 1404 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.083494 1639 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:56:00.083518 1522 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.083698 1404 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.083745 1522 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.083884 1639 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.083920 1404 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.083973 1522 maintenance_manager.cc:180] Unregistered op FlushMRSOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084095 1522 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084183 1404 maintenance_manager.cc:180] Unregistered op FlushMRSOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084178 1639 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084324 1522 maintenance_manager.cc:180] Unregistered op LogGCOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084394 1404 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084437 1639 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084470 1522 raft_consensus.cc:1495] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:56:00.084564 1404 maintenance_manager.cc:180] Unregistered op LogGCOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084691 1639 maintenance_manager.cc:180] Unregistered op FlushMRSOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.084776 1522 raft_consensus.cc:1511] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:56:00.084766 1404 raft_consensus.cc:1495] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 LEADER]: Raft consensus shutting down. I0628 23:56:00.084877 1639 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.085068 1639 maintenance_manager.cc:180] Unregistered op LogGCOp(a7f90e4fdbe34d30a37b6515985c2e34) I0628 23:56:00.085299 1639 raft_consensus.cc:1495] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:56:00.085719 1404 consensus_peers.cc:351] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 -> Peer c008181728d9485f8406ec2c378f39f7 (127.106.77.2:32804): Closing peer: c008181728d9485f8406ec2c378f39f7 I0628 23:56:00.085878 1639 raft_consensus.cc:1511] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:56:00.086501 1404 consensus_peers.cc:351] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 -> Peer ab59d0275deb4477a827b67f8fdce34f (127.106.77.1:60903): Closing peer: ab59d0275deb4477a827b67f8fdce34f I0628 23:56:00.087575 1404 raft_consensus_state.cc:377] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 LEADER]: Trying to abort 1 pending transactions. I0628 23:56:00.087764 1404 raft_consensus_state.cc:382] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 LEADER]: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6009479823112634368 op_type: NO_OP noop_request { } I0628 23:56:00.088345 1404 raft_consensus.cc:1906] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: NO_OP replication failed: Aborted: Transaction aborted I0628 23:56:00.088632 1404 raft_consensus.cc:1511] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22 [term 1 LEADER]: Raft consensus is shut down! I0628 23:56:00.090415 1522 ts_tablet_manager.cc:955] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:00.091454 1639 ts_tablet_manager.cc:955] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:00.094775 1522 ts_tablet_manager.cc:965] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Tablet deleted. Last logged OpId: 0.0 I0628 23:56:00.094877 1404 ts_tablet_manager.cc:955] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Deleting tablet data with delete state TABLET_DATA_DELETED W0628 23:56:00.095613 1522 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:56:00.095746 1522 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:56:00.095962 1522 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_wal_deleted *** Check failure stack trace: *** I0628 23:56:00.098263 1639 ts_tablet_manager.cc:965] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Tablet deleted. Last logged OpId: 0.0 W0628 23:56:00.099273 1639 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:56:00.099563 1639 fault_injection.cc:39] THIS SERVER MAY CRASH! @ 0x7f755cf31304 google::LogMessage::SendToLog() at ??:0 F0628 23:56:00.099751 1639 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_wal_deleted *** Check failure stack trace: *** I0628 23:56:00.101343 1404 ts_tablet_manager.cc:965] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Tablet deleted. Last logged OpId: 1.1 W0628 23:56:00.102339 1404 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:56:00.102442 1404 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:56:00.102581 1404 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_wal_deleted *** Check failure stack trace: *** @ 0x7f755cf31800 google::LogMessage::Flush() at ??:0 @ 0x7f32bc8b4304 google::LogMessage::SendToLog() at ??:0 @ 0x7f755cf35432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f0ff714b304 google::LogMessage::SendToLog() at ??:0 @ 0x7f32bc8b4800 google::LogMessage::Flush() at ??:0 @ 0x7f755dbfce42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7f0ff714b800 google::LogMessage::Flush() at ??:0 @ 0x7f32bc8b8432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f7564ab8e6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f0ff714f432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f32bd57fe42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7f7564b0042c kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7f0ff7e16e42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7f32c443be6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f7564b011dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7f0ffecd2e6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f32c448342c kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7f7564ade497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7f0ffed1a42c kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7f32c44841dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7f756269cef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7f0ffed1b1dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7f32c4461497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7f756269cd0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f0ffecf8497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7f32c201fef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7f75621cdd49 std::function<>::operator()() at ??:0 @ 0x7f0ffc8b6ef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7f75621cd949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7f0ffc8b6d0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f75621cf69e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f0ffc3e7d49 std::function<>::operator()() at ??:0 @ 0x7f75621d1477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f0ffc3e7949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7f75621d13db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f32c201fd0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f0ffc3e969e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f32c1b50d49 std::function<>::operator()() at ??:0 @ 0x7f75621d1384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f0ffc3eb477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f75621d11aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f0ffc3eb3db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f7562178e72 boost::function0<>::operator()() at ??:0 @ 0x7f0ffc3eb384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f32c1b50949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7f755dc67560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7f0ffc3eb1aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f32c1b5269e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f755f6a6182 start_thread at ??:0 @ 0x7f0ffc392e72 boost::function0<>::operator()() at ??:0 @ 0x7f755b23547d clone at ??:0 @ 0x7f32c1b54477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f0ff7e81560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ (nil) (unknown) @ 0x7f32c1b543db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f0ff98c0182 start_thread at ??:0 @ 0x7f0ff544f47d clone at ??:0 @ (nil) (unknown) @ 0x7f32c1b54384 boost::_bi::bind_t<>::operator()() at ??:0 W0628 23:56:00.236039 1755 catalog_manager.cc:1957] TS ab59d0275deb4477a827b67f8fdce34f: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Recv() got EOF from remote (error 108) I0628 23:56:00.237573 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=ab59d0275deb4477a827b67f8fdce34f with a delay of 60ms (attempt = 1)... @ 0x7f32c1b541aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 W0628 23:56:00.250803 1755 catalog_manager.cc:1957] TS 35c51db271ff443f872fc364af318f22: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Recv() got EOF from remote (error 108) I0628 23:56:00.251034 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=35c51db271ff443f872fc364af318f22 with a delay of 54ms (attempt = 1)... @ 0x7f32c1afbe72 boost::function0<>::operator()() at ??:0 @ 0x7f32bd5ea560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7f32bf029182 start_thread at ??:0 @ 0x7f32babb847d clone at ??:0 @ (nil) (unknown) W0628 23:56:00.289309 1755 catalog_manager.cc:1957] TS c008181728d9485f8406ec2c378f39f7: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Recv() got EOF from remote (error 108) I0628 23:56:00.289470 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=c008181728d9485f8406ec2c378f39f7 with a delay of 42ms (attempt = 1)... I0628 23:56:00.298018 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:34645 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=42992 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:55050 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 W0628 23:56:00.301888 1755 catalog_manager.cc:1957] TS ab59d0275deb4477a827b67f8fdce34f: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.1:60903: connect: Connection refused (error 111) I0628 23:56:00.302165 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=ab59d0275deb4477a827b67f8fdce34f with a delay of 51ms (attempt = 2)... W0628 23:56:00.307601 1755 catalog_manager.cc:1957] TS 35c51db271ff443f872fc364af318f22: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.0:34645: connect: Connection refused (error 111) I0628 23:56:00.307734 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=35c51db271ff443f872fc364af318f22 with a delay of 75ms (attempt = 2)... W0628 23:56:00.333660 1755 catalog_manager.cc:1957] TS c008181728d9485f8406ec2c378f39f7: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.2:32804: connect: Connection refused (error 111) I0628 23:56:00.333909 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=c008181728d9485f8406ec2c378f39f7 with a delay of 38ms (attempt = 2)... W0628 23:56:00.355082 1755 catalog_manager.cc:1957] TS ab59d0275deb4477a827b67f8fdce34f: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.1:60903: connect: Connection refused (error 111) I0628 23:56:00.355252 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=ab59d0275deb4477a827b67f8fdce34f with a delay of 89ms (attempt = 3)... W0628 23:56:00.374861 1755 catalog_manager.cc:1957] TS c008181728d9485f8406ec2c378f39f7: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.2:32804: connect: Connection refused (error 111) I0628 23:56:00.375010 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=c008181728d9485f8406ec2c378f39f7 with a delay of 75ms (attempt = 3)... W0628 23:56:00.385418 1755 catalog_manager.cc:1957] TS 35c51db271ff443f872fc364af318f22: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.0:34645: connect: Connection refused (error 111) I0628 23:56:00.385551 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=35c51db271ff443f872fc364af318f22 with a delay of 76ms (attempt = 3)... I0628 23:56:00.430778 1786 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:00.431442 1786 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:00.446228 1786 tablet_server_main.cc:54] Initializing tablet server... W0628 23:56:00.446895 1755 catalog_manager.cc:1957] TS ab59d0275deb4477a827b67f8fdce34f: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.1:60903: connect: Connection refused (error 111) I0628 23:56:00.447063 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=ab59d0275deb4477a827b67f8fdce34f with a delay of 157ms (attempt = 4)... I0628 23:56:00.449519 1786 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 166351 W0628 23:56:00.451838 1755 catalog_manager.cc:1957] TS c008181728d9485f8406ec2c378f39f7: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.2:32804: connect: Connection refused (error 111) I0628 23:56:00.451990 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=c008181728d9485f8406ec2c378f39f7 with a delay of 146ms (attempt = 4)... W0628 23:56:00.463459 1755 catalog_manager.cc:1957] TS 35c51db271ff443f872fc364af318f22: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.0:34645: connect: Connection refused (error 111) I0628 23:56:00.463625 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=35c51db271ff443f872fc364af318f22 with a delay of 130ms (attempt = 4)... I0628 23:56:00.461933 1786 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0 uuid: "35c51db271ff443f872fc364af318f22" format_stamp: "Formatted at 2016-06-28 23:55:58 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:00.498248 1786 ts_tablet_manager.cc:604] Loading metadata for tablet a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:56:00.503746 1786 ts_tablet_manager.cc:936] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:56:00.509440 1786 ts_tablet_manager.cc:955] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:00.515426 1786 ts_tablet_manager.cc:965] T a7f90e4fdbe34d30a37b6515985c2e34 P 35c51db271ff443f872fc364af318f22: Tablet deleted. Last logged OpId: 1.1 I0628 23:56:00.517537 1786 tablet_server_main.cc:57] Starting tablet server... W0628 23:56:00.596053 1755 catalog_manager.cc:1957] TS 35c51db271ff443f872fc364af318f22: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.0:34645: connect: Connection refused (error 111) I0628 23:56:00.596338 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=35c51db271ff443f872fc364af318f22 with a delay of 263ms (attempt = 5)... W0628 23:56:00.600505 1755 catalog_manager.cc:1957] TS c008181728d9485f8406ec2c378f39f7: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.2:32804: connect: Connection refused (error 111) I0628 23:56:00.600726 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=c008181728d9485f8406ec2c378f39f7 with a delay of 272ms (attempt = 5)... W0628 23:56:00.606519 1755 catalog_manager.cc:1957] TS ab59d0275deb4477a827b67f8fdce34f: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.1:60903: connect: Connection refused (error 111) I0628 23:56:00.606712 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=ab59d0275deb4477a827b67f8fdce34f with a delay of 283ms (attempt = 5)... I0628 23:56:00.610141 1786 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:34645 I0628 23:56:00.611769 1786 webserver.cc:122] Starting webserver on 127.106.77.0:42992 I0628 23:56:00.612040 1786 webserver.cc:133] Document root disabled I0628 23:56:00.613682 1786 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:42992/ I0628 23:56:00.615043 1786 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:56:00.616288 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 1786 I0628 23:56:00.616999 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:60903 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_port=43509 --webserver_interface=127.106.77.1 --tserver_master_addrs=127.0.0.1:55050 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:00.622381 1786 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:00.633040 1925 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:55050 I0628 23:56:00.633509 1925 heartbeater.cc:319] Registering TS with master... I0628 23:56:00.633787 1925 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:00.635566 1317 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "35c51db271ff443f872fc364af318f22" instance_seqno: 1467158160590804 } with Master I0628 23:56:00.731951 1927 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:00.732430 1927 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:00.745218 1927 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:00.747728 1927 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 166351 I0628 23:56:00.759811 1927 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1 uuid: "ab59d0275deb4477a827b67f8fdce34f" format_stamp: "Formatted at 2016-06-28 23:55:58 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:00.794009 1927 ts_tablet_manager.cc:604] Loading metadata for tablet a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:56:00.798666 1927 ts_tablet_manager.cc:936] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:56:00.805037 1927 ts_tablet_manager.cc:955] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:00.810642 1927 ts_tablet_manager.cc:965] T a7f90e4fdbe34d30a37b6515985c2e34 P ab59d0275deb4477a827b67f8fdce34f: Tablet deleted. Last logged OpId: 0.0 I0628 23:56:00.813016 1927 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:00.868765 1862 tablet_service.cc:653] Processing DeleteTablet for tablet a7f90e4fdbe34d30a37b6515985c2e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:00 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:53404 W0628 23:56:00.871646 1755 catalog_manager.cc:2207] TS 35c51db271ff443f872fc364af318f22: delete failed for tablet a7f90e4fdbe34d30a37b6515985c2e34 because the tablet was not found. No further retry: Not found: Tablet not found: a7f90e4fdbe34d30a37b6515985c2e34 W0628 23:56:00.875032 1755 catalog_manager.cc:1957] TS c008181728d9485f8406ec2c378f39f7: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.2:32804: connect: Connection refused (error 111) I0628 23:56:00.875252 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=c008181728d9485f8406ec2c378f39f7 with a delay of 534ms (attempt = 6)... W0628 23:56:00.892848 1755 catalog_manager.cc:1957] TS ab59d0275deb4477a827b67f8fdce34f: Delete Tablet RPC failed for tablet a7f90e4fdbe34d30a37b6515985c2e34: Network error: Client connection negotiation failed: client connection to 127.106.77.1:60903: connect: Connection refused (error 111) I0628 23:56:00.893084 1755 catalog_manager.cc:2014] Scheduling retry of a7f90e4fdbe34d30a37b6515985c2e34 Delete Tablet RPC for TS=ab59d0275deb4477a827b67f8fdce34f with a delay of 519ms (attempt = 6)... I0628 23:56:00.906204 1927 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:60903 I0628 23:56:00.907817 1927 webserver.cc:122] Starting webserver on 127.106.77.1:43509 I0628 23:56:00.907932 1927 webserver.cc:133] Document root disabled I0628 23:56:00.909379 1927 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:43509/ I0628 23:56:00.910676 1927 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:56:00.913301 1927 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:00.915681 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 1927 I0628 23:56:00.916846 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:32804 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_port=42816 --webserver_interface=127.106.77.2 --tserver_master_addrs=127.0.0.1:55050 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:00.922636 2042 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:55050 I0628 23:56:00.922967 2042 heartbeater.cc:319] Registering TS with master... I0628 23:56:00.923243 2042 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:00.924912 1317 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "ab59d0275deb4477a827b67f8fdce34f" instance_seqno: 1467158160886178 } with Master I0628 23:56:01.033363 2045 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:01.033932 2045 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:01.047307 2045 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:01.050277 2045 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 166851 I0628 23:56:01.062961 2045 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2 uuid: "c008181728d9485f8406ec2c378f39f7" format_stamp: "Formatted at 2016-06-28 23:55:59 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:01.099258 2045 ts_tablet_manager.cc:604] Loading metadata for tablet a7f90e4fdbe34d30a37b6515985c2e34 I0628 23:56:01.104429 2045 ts_tablet_manager.cc:936] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:56:01.110679 2045 ts_tablet_manager.cc:955] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:01.116462 2045 ts_tablet_manager.cc:965] T a7f90e4fdbe34d30a37b6515985c2e34 P c008181728d9485f8406ec2c378f39f7: Tablet deleted. Last logged OpId: 0.0 I0628 23:56:01.118393 2045 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:01.240274 2045 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:32804 I0628 23:56:01.241650 2045 webserver.cc:122] Starting webserver on 127.106.77.2:42816 I0628 23:56:01.241760 2045 webserver.cc:133] Document root disabled I0628 23:56:01.243453 2045 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:42816/ I0628 23:56:01.244952 2045 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_1.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:56:01.245587 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 2045 I0628 23:56:01.247184 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 1299 I0628 23:56:01.247493 2045 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:01.268048 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 1786 W0628 23:56:01.269268 2161 negotiation.cc:241] Failed RPC negotiation. Trace: 0628 23:56:01.250104 (+ 0us) reactor.cc:361] Submitting negotiation task for client connection to 127.0.0.1:55050 0628 23:56:01.252283 (+ 2179us) negotiation.cc:120] Waiting for socket to connect 0628 23:56:01.253489 (+ 1206us) sasl_client.cc:174] Called SaslClient::Negotiate() 0628 23:56:01.253597 (+ 108us) sasl_client.cc:265] SASL Client: Sending NEGOTIATE request to server. 0628 23:56:01.266682 (+ 13085us) negotiation.cc:234] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.0.1:55050: BlockingRecv error: recv error: Connection reset by peer (error 104) Metrics: {"negotiator.queue_time_us":1901,"thread_start_us":1310,"threads_started":1} W0628 23:56:01.270519 2159 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:55050: Network error: Failed to ping master at 127.0.0.1:55050: Client connection negotiation failed: client connection to 127.0.0.1:55050: BlockingRecv error: recv error: Connection reset by peer (error 104) W0628 23:56:01.272367 2159 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:55050: Network error: Failed to ping master at 127.0.0.1:55050: Client connection negotiation failed: client connection to 127.0.0.1:55050: connect: Connection refused (error 111) W0628 23:56:01.274278 2159 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:55050: Network error: Failed to ping master at 127.0.0.1:55050: Client connection negotiation failed: client connection to 127.0.0.1:55050: connect: Connection refused (error 111) W0628 23:56:01.274436 2159 heartbeater.cc:283] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. W0628 23:56:01.275961 2159 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:55050: Network error: Failed to ping master at 127.0.0.1:55050: Client connection negotiation failed: client connection to 127.0.0.1:55050: connect: Connection refused (error 111) I0628 23:56:01.284096 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 1927 I0628 23:56:01.298235 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 2045 [ OK ] FaultFlags/DeleteTableDeletedParamTest.TestRollForwardDelete/1 (3410 ms) [ RUN ] FaultFlags/DeleteTableDeletedParamTest.TestRollForwardDelete/2 I0628 23:56:01.320446 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:01.434487 2163 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:01.435071 2163 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:01.447991 2163 master_main.cc:58] Initializing master server... I0628 23:56:01.450392 2163 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 166851 I0628 23:56:01.456876 2163 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:56:01.457068 2163 server_base.cc:165] Creating new FS layout I0628 23:56:01.470659 2163 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/master/instance: uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" format_stamp: "Formatted at 2016-06-28 23:56:01 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:01.483139 2163 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/master uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" format_stamp: "Formatted at 2016-06-28 23:56:01 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:01.515664 2163 master_main.cc:61] Starting Master server... I0628 23:56:01.550837 2163 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:53041 I0628 23:56:01.552223 2163 webserver.cc:122] Starting webserver on localhost:0 I0628 23:56:01.552342 2163 webserver.cc:133] Document root disabled I0628 23:56:01.553267 2163 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:56:01.554507 2163 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:60510/ I0628 23:56:01.555909 2163 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:56:01.557569 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 2163 I0628 23:56:01.558270 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:53041 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:01.582547 2206 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9: Bootstrap starting. I0628 23:56:01.586334 2206 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:01.587007 2206 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9: No blocks or log segments found. Creating new log. I0628 23:56:01.588676 2206 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:01.593304 2206 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9: No bootstrap required, opened a new log I0628 23:56:01.599197 2206 raft_consensus.cc:269] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" member_type: VOTER } I0628 23:56:01.600283 2206 raft_consensus.cc:301] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:01.600574 2206 raft_consensus.cc:499] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3c060501ac4ffd80ce7cc9abc38aa9, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:01.600987 2206 consensus_queue.cc:162] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:01.601253 2206 raft_consensus.cc:314] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:56:01.601405 2206 raft_consensus.cc:370] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:56:01.601523 2206 raft_consensus.cc:2019] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:01.603198 2206 raft_consensus.cc:1970] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.594s I0628 23:56:01.603395 2206 raft_consensus.cc:386] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" member_type: VOTER } I0628 23:56:01.605561 2206 leader_election.cc:248] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:56:01.607291 2211 raft_consensus.cc:1970] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.567s I0628 23:56:01.607623 2211 raft_consensus.cc:1859] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:56:01.610103 2211 raft_consensus.cc:464] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 1 LEADER]: Becoming Leader. State: Replica: fa3c060501ac4ffd80ce7cc9abc38aa9, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:01.610998 2211 consensus_queue.cc:145] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" member_type: VOTER } I0628 23:56:01.633980 2206 sys_catalog.cc:235] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" config { opid_index: -1 peers { permanent_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" member_type: VOTER } } I0628 23:56:01.634806 2206 sys_catalog.cc:238] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [sys.catalog]: This master's current role is: LEADER I0628 23:56:01.634163 2212 sys_catalog.cc:235] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" config { opid_index: -1 peers { permanent_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" member_type: VOTER } } I0628 23:56:01.635431 2212 sys_catalog.cc:238] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [sys.catalog]: This master's current role is: LEADER I0628 23:56:01.638038 2215 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:56:01.636610 2213 sys_catalog.cc:235] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa3c060501ac4ffd80ce7cc9abc38aa9. Latest consensus state: current_term: 1 leader_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" config { opid_index: -1 peers { permanent_uuid: "fa3c060501ac4ffd80ce7cc9abc38aa9" member_type: VOTER } } I0628 23:56:01.639196 2213 sys_catalog.cc:238] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [sys.catalog]: This master's current role is: LEADER I0628 23:56:01.639986 2216 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:56:01.640604 2206 sys_catalog.cc:311] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:56:01.650692 2163 master_main.cc:64] Master server successfully started. I0628 23:56:01.691370 2207 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:01.691925 2207 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:01.710757 2207 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:01.713717 2207 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 166851 I0628 23:56:01.719810 2207 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:56:01.720005 2207 server_base.cc:165] Creating new FS layout I0628 23:56:01.728137 2207 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "26c183a2e76c4034ad1b4c1451a7921c" format_stamp: "Formatted at 2016-06-28 23:56:01 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:01.746929 2207 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0 uuid: "26c183a2e76c4034ad1b4c1451a7921c" format_stamp: "Formatted at 2016-06-28 23:56:01 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:01.816519 2207 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:02.240756 2207 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:45073 I0628 23:56:02.242161 2207 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:56:02.242250 2207 webserver.cc:133] Document root disabled I0628 23:56:02.243736 2207 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:37976/ I0628 23:56:02.245069 2207 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:56:02.251019 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 2207 I0628 23:56:02.251940 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:53041 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:02.253625 2207 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:02.262776 2331 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53041 I0628 23:56:02.263114 2331 heartbeater.cc:319] Registering TS with master... I0628 23:56:02.263320 2331 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:02.265038 2183 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "26c183a2e76c4034ad1b4c1451a7921c" instance_seqno: 1467158162221259 } with Master I0628 23:56:02.361886 2335 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:02.362428 2335 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:02.375453 2335 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:02.378407 2335 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 167351 I0628 23:56:02.384644 2335 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:56:02.384835 2335 server_base.cc:165] Creating new FS layout I0628 23:56:02.392755 2335 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "f9dd4151c4f44461aa07241b8bc43ecc" format_stamp: "Formatted at 2016-06-28 23:56:02 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:02.400013 2335 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1 uuid: "f9dd4151c4f44461aa07241b8bc43ecc" format_stamp: "Formatted at 2016-06-28 23:56:02 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:02.434165 2335 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:02.530563 2335 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:52087 I0628 23:56:02.532083 2335 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:56:02.532224 2335 webserver.cc:133] Document root disabled I0628 23:56:02.533812 2335 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:45224/ I0628 23:56:02.535195 2335 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:56:02.537916 2335 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:02.540328 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 2335 I0628 23:56:02.541120 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:53041 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:02.548192 2449 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53041 I0628 23:56:02.548646 2449 heartbeater.cc:319] Registering TS with master... I0628 23:56:02.548858 2449 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:02.550079 2183 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "f9dd4151c4f44461aa07241b8bc43ecc" instance_seqno: 1467158162508590 } with Master I0628 23:56:02.660176 2452 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:02.660728 2452 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:02.674976 2452 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:02.677937 2452 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 167351 I0628 23:56:02.684185 2452 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:56:02.684401 2452 server_base.cc:165] Creating new FS layout I0628 23:56:02.692816 2452 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "848b7d0ecc574b1296ff576c1a302c0e" format_stamp: "Formatted at 2016-06-28 23:56:02 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:02.700840 2452 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2 uuid: "848b7d0ecc574b1296ff576c1a302c0e" format_stamp: "Formatted at 2016-06-28 23:56:02 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:02.734905 2452 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:02.833794 2452 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:55365 I0628 23:56:02.835235 2452 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:56:02.835340 2452 webserver.cc:133] Document root disabled I0628 23:56:02.836769 2452 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:43922/ I0628 23:56:02.838052 2452 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:56:02.840201 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 2452 I0628 23:56:02.840281 2452 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:02.860230 2566 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53041 I0628 23:56:02.860702 2566 heartbeater.cc:319] Registering TS with master... I0628 23:56:02.860970 2566 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:02.862310 2183 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "848b7d0ecc574b1296ff576c1a302c0e" instance_seqno: 1467158162809982 } with Master I0628 23:56:02.864840 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:56:02.877353 27213 delete_table-test.cc:893] Running with fault flag: fault_crash_after_cmeta_deleted W0628 23:56:02.884265 2328 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:51847 forcing change of non-runtime-safe flag fault_crash_after_cmeta_deleted I0628 23:56:02.884569 2328 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:51847 changed flags via RPC: fault_crash_after_cmeta_deleted from '0' to '1.0' W0628 23:56:02.891458 2446 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:46104 forcing change of non-runtime-safe flag fault_crash_after_cmeta_deleted I0628 23:56:02.891726 2446 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:46104 changed flags via RPC: fault_crash_after_cmeta_deleted from '0' to '1.0' W0628 23:56:02.897799 2563 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:45555 forcing change of non-runtime-safe flag fault_crash_after_cmeta_deleted I0628 23:56:02.898125 2563 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:45555 changed flags via RPC: fault_crash_after_cmeta_deleted from '0' to '1.0' I0628 23:56:02.911415 2183 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:38522: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0628 23:56:02.955163 2386 tablet_service.cc:612] Processing CreateTablet for tablet 9caf8dd9519248428d58d0d5cc33889e (table=test-workload [id=4988835b4050452884533f39b29f7ed8]), partition=range: [(), ()) I0628 23:56:02.957656 2268 tablet_service.cc:612] Processing CreateTablet for tablet 9caf8dd9519248428d58d0d5cc33889e (table=test-workload [id=4988835b4050452884533f39b29f7ed8]), partition=range: [(), ()) I0628 23:56:02.959647 2503 tablet_service.cc:612] Processing CreateTablet for tablet 9caf8dd9519248428d58d0d5cc33889e (table=test-workload [id=4988835b4050452884533f39b29f7ed8]), partition=range: [(), ()) I0628 23:56:02.973026 2386 ts_tablet_manager.cc:760] Registered tablet 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:02.982879 2268 ts_tablet_manager.cc:760] Registered tablet 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:02.982977 2503 ts_tablet_manager.cc:760] Registered tablet 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:02.984889 2586 ts_tablet_manager.cc:628] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Bootstrapping tablet I0628 23:56:02.988314 2586 tablet_bootstrap.cc:393] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Bootstrap starting. I0628 23:56:02.994218 2587 ts_tablet_manager.cc:628] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Bootstrapping tablet I0628 23:56:02.995337 2586 tablet_bootstrap.cc:553] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Time spent opening tablet: real 0.004s user 0.008s sys 0.000s I0628 23:56:02.995868 2586 tablet_bootstrap.cc:493] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: No blocks or log segments found. Creating new log. I0628 23:56:02.996929 2587 tablet_bootstrap.cc:393] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Bootstrap starting. I0628 23:56:02.997823 2586 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:02.999137 2588 ts_tablet_manager.cc:628] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Bootstrapping tablet I0628 23:56:03.000633 2587 tablet_bootstrap.cc:553] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:03.001075 2587 tablet_bootstrap.cc:493] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: No blocks or log segments found. Creating new log. I0628 23:56:03.001662 2588 tablet_bootstrap.cc:393] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Bootstrap starting. I0628 23:56:03.002998 2587 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:03.006355 2588 tablet_bootstrap.cc:553] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:03.007128 2588 tablet_bootstrap.cc:493] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: No blocks or log segments found. Creating new log. I0628 23:56:03.009537 2587 tablet_bootstrap.cc:393] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: No bootstrap required, opened a new log I0628 23:56:03.009784 2588 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:03.010092 2587 ts_tablet_manager.cc:633] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s I0628 23:56:03.011364 2586 tablet_bootstrap.cc:393] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: No bootstrap required, opened a new log I0628 23:56:03.011824 2586 ts_tablet_manager.cc:633] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.000s I0628 23:56:03.022236 2587 raft_consensus.cc:269] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f9dd4151c4f44461aa07241b8bc43ecc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 52087 } } peers { permanent_uuid: "848b7d0ecc574b1296ff576c1a302c0e" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 55365 } } peers { permanent_uuid: "26c183a2e76c4034ad1b4c1451a7921c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 45073 } } I0628 23:56:03.022861 2586 raft_consensus.cc:269] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f9dd4151c4f44461aa07241b8bc43ecc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 52087 } } peers { permanent_uuid: "848b7d0ecc574b1296ff576c1a302c0e" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 55365 } } peers { permanent_uuid: "26c183a2e76c4034ad1b4c1451a7921c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 45073 } } I0628 23:56:03.024996 2587 raft_consensus.cc:301] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:03.025174 2586 raft_consensus.cc:301] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:03.025336 2587 raft_consensus.cc:499] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c183a2e76c4034ad1b4c1451a7921c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:03.025512 2586 raft_consensus.cc:499] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dd4151c4f44461aa07241b8bc43ecc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:03.026111 2586 consensus_queue.cc:162] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:03.026062 2587 consensus_queue.cc:162] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:03.031374 2586 ts_tablet_manager.cc:659] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Time spent starting tablet: real 0.019s user 0.012s sys 0.008s I0628 23:56:03.032563 2587 ts_tablet_manager.cc:659] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Time spent starting tablet: real 0.022s user 0.016s sys 0.000s I0628 23:56:03.033308 2588 tablet_bootstrap.cc:393] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: No bootstrap required, opened a new log I0628 23:56:03.033788 2588 ts_tablet_manager.cc:633] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.020s I0628 23:56:03.043427 2588 raft_consensus.cc:269] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "f9dd4151c4f44461aa07241b8bc43ecc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 52087 } } peers { permanent_uuid: "848b7d0ecc574b1296ff576c1a302c0e" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 55365 } } peers { permanent_uuid: "26c183a2e76c4034ad1b4c1451a7921c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 45073 } } I0628 23:56:03.045617 2588 raft_consensus.cc:301] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:03.045855 2588 raft_consensus.cc:499] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848b7d0ecc574b1296ff576c1a302c0e, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:03.046331 2588 consensus_queue.cc:162] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:03.050220 2588 ts_tablet_manager.cc:659] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s I0628 23:56:03.478513 2600 raft_consensus.cc:370] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:56:03.478703 2600 raft_consensus.cc:2019] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:03.480651 2600 raft_consensus.cc:1970] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.590s I0628 23:56:03.480845 2600 raft_consensus.cc:386] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "f9dd4151c4f44461aa07241b8bc43ecc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 52087 } } peers { permanent_uuid: "848b7d0ecc574b1296ff576c1a302c0e" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 55365 } } peers { permanent_uuid: "26c183a2e76c4034ad1b4c1451a7921c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 45073 } } I0628 23:56:03.484002 2600 leader_election.cc:223] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [CANDIDATE]: Term 1 election: Requesting vote from peer f9dd4151c4f44461aa07241b8bc43ecc I0628 23:56:03.484609 2600 leader_election.cc:223] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [CANDIDATE]: Term 1 election: Requesting vote from peer 26c183a2e76c4034ad1b4c1451a7921c I0628 23:56:03.502153 2406 raft_consensus.cc:2019] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:03.502164 2288 raft_consensus.cc:2019] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:03.504117 2288 raft_consensus.cc:1970] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.641s I0628 23:56:03.504232 2406 raft_consensus.cc:1970] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.645s I0628 23:56:03.506013 2288 raft_consensus.cc:1677] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 848b7d0ecc574b1296ff576c1a302c0e in term 1. I0628 23:56:03.506726 2406 raft_consensus.cc:1677] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 848b7d0ecc574b1296ff576c1a302c0e in term 1. I0628 23:56:03.507426 2457 leader_election.cc:347] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [CANDIDATE]: Term 1 election: Vote granted by peer 26c183a2e76c4034ad1b4c1451a7921c I0628 23:56:03.507686 2457 leader_election.cc:248] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:56:03.507911 2459 leader_election.cc:347] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [CANDIDATE]: Term 1 election: Vote granted by peer f9dd4151c4f44461aa07241b8bc43ecc I0628 23:56:03.508319 2601 raft_consensus.cc:1970] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.557s I0628 23:56:03.508596 2601 raft_consensus.cc:1859] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:56:03.510396 2601 raft_consensus.cc:464] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 LEADER]: Becoming Leader. State: Replica: 848b7d0ecc574b1296ff576c1a302c0e, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:03.511206 2601 consensus_queue.cc:145] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "f9dd4151c4f44461aa07241b8bc43ecc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 52087 } } peers { permanent_uuid: "848b7d0ecc574b1296ff576c1a302c0e" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 55365 } } peers { permanent_uuid: "26c183a2e76c4034ad1b4c1451a7921c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 45073 } } W0628 23:56:03.539737 2609 raft_consensus_state.cc:541] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:56:03.539263 2182 catalog_manager.cc:1673] Tablet: 9caf8dd9519248428d58d0d5cc33889e reported consensus state change. New consensus state: current_term: 1 leader_uuid: "848b7d0ecc574b1296ff576c1a302c0e" config { opid_index: -1 peers { permanent_uuid: "f9dd4151c4f44461aa07241b8bc43ecc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 52087 } } peers { permanent_uuid: "848b7d0ecc574b1296ff576c1a302c0e" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 55365 } } peers { permanent_uuid: "26c183a2e76c4034ad1b4c1451a7921c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 45073 } } } I0628 23:56:03.667345 2182 catalog_manager.cc:1036] Servicing DeleteTable request from {real_user=slave, eff_user=} at 127.0.0.1:38518: table { table_name: "test-workload" } I0628 23:56:03.673259 2182 catalog_manager.cc:2531] Sending DeleteTablet for 3 replicas of tablet 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:03.673377 2182 catalog_manager.cc:2547] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9: Deleting tablet 9caf8dd9519248428d58d0d5cc33889e on peer f9dd4151c4f44461aa07241b8bc43ecc with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:03 UTC) I0628 23:56:03.673833 2182 catalog_manager.cc:2547] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9: Deleting tablet 9caf8dd9519248428d58d0d5cc33889e on peer 848b7d0ecc574b1296ff576c1a302c0e with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:03 UTC) I0628 23:56:03.674263 2182 catalog_manager.cc:2547] T 00000000000000000000000000000000 P fa3c060501ac4ffd80ce7cc9abc38aa9: Deleting tablet 9caf8dd9519248428d58d0d5cc33889e on peer 26c183a2e76c4034ad1b4c1451a7921c with delete type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:03 UTC) I0628 23:56:03.674777 2182 catalog_manager.cc:1119] Successfully deleted table test-workload [id=4988835b4050452884533f39b29f7ed8] per request from {real_user=slave, eff_user=} at 127.0.0.1:38518 I0628 23:56:03.675038 2386 tablet_service.cc:653] Processing DeleteTablet for tablet 9caf8dd9519248428d58d0d5cc33889e with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:03 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:46107 I0628 23:56:03.675490 2386 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:03.675282 2503 tablet_service.cc:653] Processing DeleteTablet for tablet 9caf8dd9519248428d58d0d5cc33889e with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:03 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:45559 I0628 23:56:03.675796 2503 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:03.675936 2386 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.675964 2268 tablet_service.cc:653] Processing DeleteTablet for tablet 9caf8dd9519248428d58d0d5cc33889e with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:03 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:51852 I0628 23:56:03.676235 2503 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676280 2386 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676375 2268 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:03.676489 2503 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676512 2386 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676708 2386 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676697 2503 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676750 2268 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676820 2386 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676924 2386 maintenance_manager.cc:180] Unregistered op LogGCOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.676946 2503 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.677019 2268 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.677067 2386 raft_consensus.cc:1495] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:56:03.677140 2503 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.677250 2268 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.677322 2503 maintenance_manager.cc:180] Unregistered op LogGCOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.677403 2386 raft_consensus.cc:1511] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:56:03.677503 2268 maintenance_manager.cc:180] Unregistered op FlushMRSOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.677521 2503 raft_consensus.cc:1495] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 LEADER]: Raft consensus shutting down. I0628 23:56:03.677686 2268 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.677852 2268 maintenance_manager.cc:180] Unregistered op LogGCOp(9caf8dd9519248428d58d0d5cc33889e) I0628 23:56:03.678050 2268 raft_consensus.cc:1495] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:56:03.678385 2503 consensus_peers.cc:351] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e -> Peer 26c183a2e76c4034ad1b4c1451a7921c (127.106.77.0:45073): Closing peer: 26c183a2e76c4034ad1b4c1451a7921c I0628 23:56:03.678439 2268 raft_consensus.cc:1511] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:56:03.679030 2503 consensus_peers.cc:351] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e -> Peer f9dd4151c4f44461aa07241b8bc43ecc (127.106.77.1:52087): Closing peer: f9dd4151c4f44461aa07241b8bc43ecc I0628 23:56:03.679868 2503 raft_consensus_state.cc:377] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 LEADER]: Trying to abort 1 pending transactions. I0628 23:56:03.680037 2503 raft_consensus_state.cc:382] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 LEADER]: Aborting transaction as it isn't in flight: id { term: 1 index: 1 } timestamp: 6009479837760675840 op_type: NO_OP noop_request { } I0628 23:56:03.680686 2503 raft_consensus.cc:1906] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: NO_OP replication failed: Aborted: Transaction aborted I0628 23:56:03.680972 2503 raft_consensus.cc:1511] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e [term 1 LEADER]: Raft consensus is shut down! I0628 23:56:03.681821 2386 ts_tablet_manager.cc:955] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:03.684234 2268 ts_tablet_manager.cc:955] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:03.686352 2386 ts_tablet_manager.cc:965] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Tablet deleted. Last logged OpId: 0.0 W0628 23:56:03.687242 2386 fault_injection.cc:38] FAULT INJECTION ENABLED! I0628 23:56:03.687167 2503 ts_tablet_manager.cc:955] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Deleting tablet data with delete state TABLET_DATA_DELETED W0628 23:56:03.687341 2386 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:56:03.687513 2386 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_cmeta_deleted *** Check failure stack trace: *** I0628 23:56:03.690229 2268 ts_tablet_manager.cc:965] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Tablet deleted. Last logged OpId: 0.0 @ 0x7fe7b87bd304 google::LogMessage::SendToLog() at ??:0 W0628 23:56:03.691229 2268 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:56:03.691339 2268 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:56:03.691494 2268 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_cmeta_deleted *** Check failure stack trace: *** @ 0x7fe7b87bd800 google::LogMessage::Flush() at ??:0 I0628 23:56:03.694305 2503 ts_tablet_manager.cc:965] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Tablet deleted. Last logged OpId: 1.1 W0628 23:56:03.695433 2503 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:56:03.695540 2503 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:56:03.695679 2503 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_cmeta_deleted *** Check failure stack trace: *** @ 0x7f18d8672304 google::LogMessage::SendToLog() at ??:0 @ 0x7fe7b87c1432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7fe7b9488e42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7f8cf5c58304 google::LogMessage::SendToLog() at ??:0 @ 0x7fe7c0344e6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f18d8672800 google::LogMessage::Flush() at ??:0 @ 0x7f8cf5c58800 google::LogMessage::Flush() at ??:0 @ 0x7fe7c038c4a3 kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7f18d8676432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7fe7c038d1dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7f8cf5c5c432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f18d933de42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7fe7c036a497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7f8cf6923e42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7fe7bdf28ef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7f18e01f9e6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7fe7bdf28d0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f8cfd7dfe6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f18e02414a3 kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7fe7bda59d49 std::function<>::operator()() at ??:0 @ 0x7f8cfd8274a3 kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7fe7bda59949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7fe7bda5b69e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f8cfd8281dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7fe7bda5d477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f18e02421dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7f8cfd805497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7fe7bda5d3db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f8cfb3c3ef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7fe7bda5d384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f18e021f497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7f8cfb3c3d0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7fe7bda5d1aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f18dddddef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7f8cfaef4d49 std::function<>::operator()() at ??:0 @ 0x7fe7bda04e72 boost::function0<>::operator()() at ??:0 @ 0x7f18dddddd0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7fe7b94f3560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7f18dd90ed49 std::function<>::operator()() at ??:0 @ 0x7f8cfaef4949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7fe7baf32182 start_thread at ??:0 @ 0x7f8cfaef669e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f18dd90e949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7fe7b6ac147d clone at ??:0 @ 0x7f18dd91069e kudu::rpc::ServicePool::RunThread() at ??:0 @ (nil) (unknown) @ 0x7f18dd912477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f8cfaef8477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f18dd9123db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f8cfaef83db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f8cfaef8384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f18dd912384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f8cfaef81aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f18dd9121aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f8cfae9fe72 boost::function0<>::operator()() at ??:0 @ 0x7f18dd8b9e72 boost::function0<>::operator()() at ??:0 @ 0x7f8cf698e560 kudu::Thread::SuperviseThread() at ??:0 @ 0x7f18d93a8560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7f8cf83cd182 start_thread at ??:0 @ 0x7f18dade7182 start_thread at ??:0 W0628 23:56:03.818742 2621 catalog_manager.cc:1957] TS f9dd4151c4f44461aa07241b8bc43ecc: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Recv() got EOF from remote (error 108) I0628 23:56:03.818987 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=f9dd4151c4f44461aa07241b8bc43ecc with a delay of 60ms (attempt = 1)... @ 0x7f8cf3f5c47d clone at ??:0 @ 0x7f18d697647d clone at ??:0 @ (nil) (unknown) @ (nil) (unknown) W0628 23:56:03.844728 2621 catalog_manager.cc:1957] TS 848b7d0ecc574b1296ff576c1a302c0e: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Recv() got EOF from remote (error 108) I0628 23:56:03.845016 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=848b7d0ecc574b1296ff576c1a302c0e with a delay of 54ms (attempt = 1)... W0628 23:56:03.847308 2621 catalog_manager.cc:1957] TS 26c183a2e76c4034ad1b4c1451a7921c: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Recv() got EOF from remote (error 108) I0628 23:56:03.847455 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=26c183a2e76c4034ad1b4c1451a7921c with a delay of 42ms (attempt = 1)... W0628 23:56:03.882746 2621 catalog_manager.cc:1957] TS f9dd4151c4f44461aa07241b8bc43ecc: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.1:52087: connect: Connection refused (error 111) I0628 23:56:03.882896 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=f9dd4151c4f44461aa07241b8bc43ecc with a delay of 51ms (attempt = 2)... I0628 23:56:03.888627 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:45073 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=37976 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:53041 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 W0628 23:56:03.891190 2621 catalog_manager.cc:1957] TS 26c183a2e76c4034ad1b4c1451a7921c: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.0:45073: connect: Connection refused (error 111) I0628 23:56:03.891408 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=26c183a2e76c4034ad1b4c1451a7921c with a delay of 75ms (attempt = 2)... W0628 23:56:03.900707 2621 catalog_manager.cc:1957] TS 848b7d0ecc574b1296ff576c1a302c0e: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.2:55365: connect: Connection refused (error 111) I0628 23:56:03.900895 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=848b7d0ecc574b1296ff576c1a302c0e with a delay of 38ms (attempt = 2)... W0628 23:56:03.936573 2621 catalog_manager.cc:1957] TS f9dd4151c4f44461aa07241b8bc43ecc: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.1:52087: connect: Connection refused (error 111) I0628 23:56:03.936780 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=f9dd4151c4f44461aa07241b8bc43ecc with a delay of 89ms (attempt = 3)... W0628 23:56:03.941279 2621 catalog_manager.cc:1957] TS 848b7d0ecc574b1296ff576c1a302c0e: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.2:55365: connect: Connection refused (error 111) I0628 23:56:03.941409 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=848b7d0ecc574b1296ff576c1a302c0e with a delay of 75ms (attempt = 3)... W0628 23:56:03.968612 2621 catalog_manager.cc:1957] TS 26c183a2e76c4034ad1b4c1451a7921c: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.0:45073: connect: Connection refused (error 111) I0628 23:56:03.968750 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=26c183a2e76c4034ad1b4c1451a7921c with a delay of 76ms (attempt = 3)... I0628 23:56:04.013828 2654 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:04.014297 2654 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB W0628 23:56:04.018595 2621 catalog_manager.cc:1957] TS 848b7d0ecc574b1296ff576c1a302c0e: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.2:55365: connect: Connection refused (error 111) I0628 23:56:04.018754 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=848b7d0ecc574b1296ff576c1a302c0e with a delay of 157ms (attempt = 4)... W0628 23:56:04.027832 2621 catalog_manager.cc:1957] TS f9dd4151c4f44461aa07241b8bc43ecc: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.1:52087: connect: Connection refused (error 111) I0628 23:56:04.028074 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=f9dd4151c4f44461aa07241b8bc43ecc with a delay of 146ms (attempt = 4)... I0628 23:56:04.029572 2654 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:04.032292 2654 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 168351 I0628 23:56:04.043609 2654 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0 uuid: "26c183a2e76c4034ad1b4c1451a7921c" format_stamp: "Formatted at 2016-06-28 23:56:01 on dist-test-slave-dist-test-slave-cgs5" W0628 23:56:04.046947 2621 catalog_manager.cc:1957] TS 26c183a2e76c4034ad1b4c1451a7921c: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.0:45073: connect: Connection refused (error 111) I0628 23:56:04.047096 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=26c183a2e76c4034ad1b4c1451a7921c with a delay of 130ms (attempt = 4)... I0628 23:56:04.078824 2654 ts_tablet_manager.cc:604] Loading metadata for tablet 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:04.083614 2654 ts_tablet_manager.cc:936] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:56:04.089612 2654 ts_tablet_manager.cc:955] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:04.095130 2654 ts_tablet_manager.cc:965] T 9caf8dd9519248428d58d0d5cc33889e P 26c183a2e76c4034ad1b4c1451a7921c: Tablet deleted. Last logged OpId: 0.0 I0628 23:56:04.096899 2654 tablet_server_main.cc:57] Starting tablet server... W0628 23:56:04.176805 2621 catalog_manager.cc:1957] TS f9dd4151c4f44461aa07241b8bc43ecc: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.1:52087: connect: Connection refused (error 111) I0628 23:56:04.177052 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=f9dd4151c4f44461aa07241b8bc43ecc with a delay of 263ms (attempt = 5)... W0628 23:56:04.178825 2621 catalog_manager.cc:1957] TS 848b7d0ecc574b1296ff576c1a302c0e: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.2:55365: connect: Connection refused (error 111) I0628 23:56:04.179029 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=848b7d0ecc574b1296ff576c1a302c0e with a delay of 272ms (attempt = 5)... W0628 23:56:04.180486 2621 catalog_manager.cc:1957] TS 26c183a2e76c4034ad1b4c1451a7921c: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.0:45073: connect: Connection refused (error 111) I0628 23:56:04.180730 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=26c183a2e76c4034ad1b4c1451a7921c with a delay of 283ms (attempt = 5)... I0628 23:56:04.197619 2654 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:45073 I0628 23:56:04.199062 2654 webserver.cc:122] Starting webserver on 127.106.77.0:37976 I0628 23:56:04.199149 2654 webserver.cc:133] Document root disabled I0628 23:56:04.200592 2654 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:37976/ I0628 23:56:04.201858 2654 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:56:04.204262 2654 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:04.206676 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 2654 I0628 23:56:04.207689 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:52087 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_port=45224 --webserver_interface=127.106.77.1 --tserver_master_addrs=127.0.0.1:53041 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:04.215114 2789 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53041 I0628 23:56:04.215565 2789 heartbeater.cc:319] Registering TS with master... I0628 23:56:04.215838 2789 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:04.217628 2182 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "26c183a2e76c4034ad1b4c1451a7921c" instance_seqno: 1467158164168860 } with Master I0628 23:56:04.323963 2792 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:04.324466 2792 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:04.338172 2792 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:04.340853 2792 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 168351 I0628 23:56:04.352159 2792 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1 uuid: "f9dd4151c4f44461aa07241b8bc43ecc" format_stamp: "Formatted at 2016-06-28 23:56:02 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:04.386636 2792 ts_tablet_manager.cc:604] Loading metadata for tablet 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:04.392642 2792 ts_tablet_manager.cc:936] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:56:04.398459 2792 ts_tablet_manager.cc:955] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:04.404232 2792 ts_tablet_manager.cc:965] T 9caf8dd9519248428d58d0d5cc33889e P f9dd4151c4f44461aa07241b8bc43ecc: Tablet deleted. Last logged OpId: 0.0 I0628 23:56:04.405978 2792 tablet_server_main.cc:57] Starting tablet server... W0628 23:56:04.443619 2621 catalog_manager.cc:1957] TS f9dd4151c4f44461aa07241b8bc43ecc: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.1:52087: connect: Connection refused (error 111) I0628 23:56:04.443847 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=f9dd4151c4f44461aa07241b8bc43ecc with a delay of 534ms (attempt = 6)... W0628 23:56:04.453202 2621 catalog_manager.cc:1957] TS 848b7d0ecc574b1296ff576c1a302c0e: Delete Tablet RPC failed for tablet 9caf8dd9519248428d58d0d5cc33889e: Network error: Client connection negotiation failed: client connection to 127.106.77.2:55365: connect: Connection refused (error 111) I0628 23:56:04.453418 2621 catalog_manager.cc:2014] Scheduling retry of 9caf8dd9519248428d58d0d5cc33889e Delete Tablet RPC for TS=848b7d0ecc574b1296ff576c1a302c0e with a delay of 519ms (attempt = 6)... I0628 23:56:04.473244 2726 tablet_service.cc:653] Processing DeleteTablet for tablet 9caf8dd9519248428d58d0d5cc33889e with delete_type TABLET_DATA_DELETED (Table deleted at 2016-06-28 23:56:03 UTC) from {real_user=slave, eff_user=} at 127.0.0.1:51868 W0628 23:56:04.475759 2621 catalog_manager.cc:2207] TS 26c183a2e76c4034ad1b4c1451a7921c: delete failed for tablet 9caf8dd9519248428d58d0d5cc33889e because the tablet was not found. No further retry: Not found: Tablet not found: 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:04.497519 2792 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:52087 I0628 23:56:04.499028 2792 webserver.cc:122] Starting webserver on 127.106.77.1:45224 I0628 23:56:04.499119 2792 webserver.cc:133] Document root disabled I0628 23:56:04.500553 2792 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:45224/ I0628 23:56:04.501788 2792 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:56:04.503849 2792 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:04.508035 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 2792 I0628 23:56:04.509317 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:55365 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_port=43922 --webserver_interface=127.106.77.2 --tserver_master_addrs=127.0.0.1:53041 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:04.513633 2906 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:53041 I0628 23:56:04.514034 2906 heartbeater.cc:319] Registering TS with master... I0628 23:56:04.514305 2906 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:04.515827 2182 ts_manager.cc:87] Re-registered known tablet server { permanent_uuid: "f9dd4151c4f44461aa07241b8bc43ecc" instance_seqno: 1467158164479459 } with Master I0628 23:56:04.621737 2909 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:04.622231 2909 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:04.635097 2909 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:04.637925 2909 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 168351 I0628 23:56:04.649421 2909 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2 uuid: "848b7d0ecc574b1296ff576c1a302c0e" format_stamp: "Formatted at 2016-06-28 23:56:02 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:04.683769 2909 ts_tablet_manager.cc:604] Loading metadata for tablet 9caf8dd9519248428d58d0d5cc33889e I0628 23:56:04.688822 2909 ts_tablet_manager.cc:936] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED I0628 23:56:04.694344 2909 ts_tablet_manager.cc:955] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:04.700036 2909 ts_tablet_manager.cc:965] T 9caf8dd9519248428d58d0d5cc33889e P 848b7d0ecc574b1296ff576c1a302c0e: Tablet deleted. Last logged OpId: 1.1 I0628 23:56:04.701653 2909 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:04.791910 2909 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:55365 I0628 23:56:04.793498 2909 webserver.cc:122] Starting webserver on 127.106.77.2:43922 I0628 23:56:04.793591 2909 webserver.cc:133] Document root disabled I0628 23:56:04.795202 2909 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:43922/ I0628 23:56:04.796545 2909 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableDeletedParamTest.TestRollForwardDelete_2.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:56:04.798652 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 2909 I0628 23:56:04.798939 2909 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:04.800277 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 2163 W0628 23:56:04.819000 3025 negotiation.cc:241] Failed RPC negotiation. Trace: 0628 23:56:04.800985 (+ 0us) reactor.cc:361] Submitting negotiation task for client connection to 127.0.0.1:53041 0628 23:56:04.802912 (+ 1927us) negotiation.cc:120] Waiting for socket to connect 0628 23:56:04.803923 (+ 1011us) sasl_client.cc:174] Called SaslClient::Negotiate() 0628 23:56:04.804031 (+ 108us) sasl_client.cc:265] SASL Client: Sending NEGOTIATE request to server. 0628 23:56:04.817340 (+ 13309us) negotiation.cc:234] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.0.1:53041: BlockingRecv error: recv error: Connection reset by peer (error 104) Metrics: {"negotiator.queue_time_us":1673,"thread_start_us":1159,"threads_started":1} W0628 23:56:04.820039 3023 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:53041: Network error: Failed to ping master at 127.0.0.1:53041: Client connection negotiation failed: client connection to 127.0.0.1:53041: BlockingRecv error: recv error: Connection reset by peer (error 104) I0628 23:56:04.820523 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 2654 W0628 23:56:04.833575 3023 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:53041: Network error: Failed to ping master at 127.0.0.1:53041: Client connection negotiation failed: client connection to 127.0.0.1:53041: connect: Connection refused (error 111) W0628 23:56:04.835577 3023 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:53041: Network error: Failed to ping master at 127.0.0.1:53041: Client connection negotiation failed: client connection to 127.0.0.1:53041: connect: Connection refused (error 111) I0628 23:56:04.835798 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 2792 W0628 23:56:04.835784 3023 heartbeater.cc:283] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. W0628 23:56:04.837481 3023 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:53041: Network error: Failed to ping master at 127.0.0.1:53041: Client connection negotiation failed: client connection to 127.0.0.1:53041: connect: Connection refused (error 111) I0628 23:56:04.848608 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 2909 [ OK ] FaultFlags/DeleteTableDeletedParamTest.TestRollForwardDelete/2 (3549 ms) [----------] 3 tests from FaultFlags/DeleteTableDeletedParamTest (10211 ms total) [----------] 2 tests from FaultFlags/DeleteTableTombstonedParamTest [ RUN ] FaultFlags/DeleteTableTombstonedParamTest.TestTabletTombstone/0 I0628 23:56:04.869912 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:04.982094 3027 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:04.982561 3027 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:04.994685 3027 master_main.cc:58] Initializing master server... I0628 23:56:04.997516 3027 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 168351 I0628 23:56:05.003949 3027 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:56:05.004181 3027 server_base.cc:165] Creating new FS layout I0628 23:56:05.017556 3027 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/master/instance: uuid: "8db71b66c22d43f9986e1f23080e9c76" format_stamp: "Formatted at 2016-06-28 23:56:05 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:05.029986 3027 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/master uuid: "8db71b66c22d43f9986e1f23080e9c76" format_stamp: "Formatted at 2016-06-28 23:56:05 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:05.061542 3027 master_main.cc:61] Starting Master server... I0628 23:56:05.095038 3027 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:58376 I0628 23:56:05.096393 3027 webserver.cc:122] Starting webserver on localhost:0 I0628 23:56:05.096492 3027 webserver.cc:133] Document root disabled I0628 23:56:05.097321 3027 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:56:05.098733 3027 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:53725/ I0628 23:56:05.100183 3027 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:56:05.106873 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 3027 I0628 23:56:05.107725 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:58376 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0 --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:05.125638 3070 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76: Bootstrap starting. I0628 23:56:05.129556 3070 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:05.130089 3070 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76: No blocks or log segments found. Creating new log. I0628 23:56:05.131712 3070 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:05.136306 3070 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76: No bootstrap required, opened a new log I0628 23:56:05.142385 3070 raft_consensus.cc:269] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "8db71b66c22d43f9986e1f23080e9c76" member_type: VOTER } I0628 23:56:05.143493 3070 raft_consensus.cc:301] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:05.143749 3070 raft_consensus.cc:499] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db71b66c22d43f9986e1f23080e9c76, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:05.144119 3070 consensus_queue.cc:162] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:05.144443 3070 raft_consensus.cc:314] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:56:05.144610 3070 raft_consensus.cc:370] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:56:05.144757 3070 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:05.146415 3070 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.610s I0628 23:56:05.146610 3070 raft_consensus.cc:386] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "8db71b66c22d43f9986e1f23080e9c76" member_type: VOTER } I0628 23:56:05.148602 3070 leader_election.cc:248] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:56:05.150169 3075 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.617s I0628 23:56:05.150511 3075 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:56:05.152598 3075 raft_consensus.cc:464] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 1 LEADER]: Becoming Leader. State: Replica: 8db71b66c22d43f9986e1f23080e9c76, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:05.153419 3075 consensus_queue.cc:145] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8db71b66c22d43f9986e1f23080e9c76" member_type: VOTER } I0628 23:56:05.176730 3070 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "8db71b66c22d43f9986e1f23080e9c76" config { opid_index: -1 peers { permanent_uuid: "8db71b66c22d43f9986e1f23080e9c76" member_type: VOTER } } I0628 23:56:05.177235 3070 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [sys.catalog]: This master's current role is: LEADER I0628 23:56:05.177760 3076 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8db71b66c22d43f9986e1f23080e9c76" config { opid_index: -1 peers { permanent_uuid: "8db71b66c22d43f9986e1f23080e9c76" member_type: VOTER } } I0628 23:56:05.181253 3076 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [sys.catalog]: This master's current role is: LEADER I0628 23:56:05.182020 3079 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:56:05.182580 3070 sys_catalog.cc:311] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [sys.catalog]: configured and running, proceeding with master startup. I0628 23:56:05.184056 3027 master_main.cc:64] Master server successfully started. I0628 23:56:05.184634 3078 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:56:05.179188 3077 sys_catalog.cc:235] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db71b66c22d43f9986e1f23080e9c76. Latest consensus state: current_term: 1 leader_uuid: "8db71b66c22d43f9986e1f23080e9c76" config { opid_index: -1 peers { permanent_uuid: "8db71b66c22d43f9986e1f23080e9c76" member_type: VOTER } } I0628 23:56:05.188073 3077 sys_catalog.cc:238] T 00000000000000000000000000000000 P 8db71b66c22d43f9986e1f23080e9c76 [sys.catalog]: This master's current role is: LEADER I0628 23:56:05.230686 3071 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:05.231209 3071 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:05.250671 3071 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:05.253582 3071 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 168851 I0628 23:56:05.259384 3071 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:56:05.259565 3071 server_base.cc:165] Creating new FS layout I0628 23:56:05.268337 3071 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "2b0c1f965f1d485dbf65b77be7285067" format_stamp: "Formatted at 2016-06-28 23:56:05 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:05.286934 3071 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0 uuid: "2b0c1f965f1d485dbf65b77be7285067" format_stamp: "Formatted at 2016-06-28 23:56:05 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:05.356794 3071 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:05.692479 3071 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:33736 I0628 23:56:05.693815 3071 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:56:05.693913 3071 webserver.cc:133] Document root disabled I0628 23:56:05.695344 3071 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:35234/ I0628 23:56:05.696753 3071 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:56:05.698649 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 3071 I0628 23:56:05.699502 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:58376 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1 --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:05.717910 3071 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:05.722051 3195 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:58376 I0628 23:56:05.722381 3195 heartbeater.cc:319] Registering TS with master... I0628 23:56:05.722579 3195 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:05.724740 3047 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" instance_seqno: 1467158165663664 } with Master I0628 23:56:05.813103 3198 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:05.813575 3198 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:05.826119 3198 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:05.828805 3198 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 168851 I0628 23:56:05.834730 3198 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:56:05.834926 3198 server_base.cc:165] Creating new FS layout I0628 23:56:05.843041 3198 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "12be474fbcef4f68b20a52be2d4796cc" format_stamp: "Formatted at 2016-06-28 23:56:05 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:05.851197 3198 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1 uuid: "12be474fbcef4f68b20a52be2d4796cc" format_stamp: "Formatted at 2016-06-28 23:56:05 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:05.884845 3198 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:05.976248 3198 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:56466 I0628 23:56:05.978005 3198 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:56:05.978123 3198 webserver.cc:133] Document root disabled I0628 23:56:05.979612 3198 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:50309/ I0628 23:56:05.980965 3198 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:56:05.983131 3198 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:05.987078 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 3198 I0628 23:56:05.987826 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:58376 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2 --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:05.991957 3313 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:58376 I0628 23:56:05.992314 3313 heartbeater.cc:319] Registering TS with master... I0628 23:56:05.992538 3313 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:05.994010 3047 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" instance_seqno: 1467158165957353 } with Master I0628 23:56:06.098348 3316 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:06.098974 3316 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:06.112308 3316 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:06.114938 3316 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 169351 I0628 23:56:06.121114 3316 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:56:06.121345 3316 server_base.cc:165] Creating new FS layout I0628 23:56:06.129287 3316 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "a618b6bf22d84b37b5540eb843f950e0" format_stamp: "Formatted at 2016-06-28 23:56:06 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:06.136971 3316 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2 uuid: "a618b6bf22d84b37b5540eb843f950e0" format_stamp: "Formatted at 2016-06-28 23:56:06 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:06.170848 3316 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:06.260257 3316 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:53218 I0628 23:56:06.261838 3316 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:56:06.261932 3316 webserver.cc:133] Document root disabled I0628 23:56:06.263466 3316 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:36665/ I0628 23:56:06.264936 3316 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:56:06.265524 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 3316 I0628 23:56:06.268138 3316 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:06.275420 3430 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:58376 I0628 23:56:06.275781 3430 heartbeater.cc:319] Registering TS with master... I0628 23:56:06.276058 3430 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:06.277288 3047 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" instance_seqno: 1467158166242222 } with Master I0628 23:56:06.278901 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:56:06.291473 27213 delete_table-test.cc:948] Running with fault flag: fault_crash_after_blocks_deleted I0628 23:56:06.294097 3046 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:32900: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\377\377\377?" } partition_schema { range_schema { columns { name: "key" } } } I0628 23:56:06.337458 3132 tablet_service.cc:612] Processing CreateTablet for tablet 8f3895bbb21340afbb5b7d734de4db9d (table=test-workload [id=12ac1e914ab14b26b2b839bbd0dc38b4]), partition=range: [(), (int32 key=1073741823)) I0628 23:56:06.337456 3250 tablet_service.cc:612] Processing CreateTablet for tablet 8f3895bbb21340afbb5b7d734de4db9d (table=test-workload [id=12ac1e914ab14b26b2b839bbd0dc38b4]), partition=range: [(), (int32 key=1073741823)) I0628 23:56:06.349496 3249 tablet_service.cc:612] Processing CreateTablet for tablet df3659ee08ea4a5fa181023ae31550b7 (table=test-workload [id=12ac1e914ab14b26b2b839bbd0dc38b4]), partition=range: [(int32 key=1073741823), ()) I0628 23:56:06.351318 3131 tablet_service.cc:612] Processing CreateTablet for tablet df3659ee08ea4a5fa181023ae31550b7 (table=test-workload [id=12ac1e914ab14b26b2b839bbd0dc38b4]), partition=range: [(int32 key=1073741823), ()) I0628 23:56:06.351604 3367 tablet_service.cc:612] Processing CreateTablet for tablet 8f3895bbb21340afbb5b7d734de4db9d (table=test-workload [id=12ac1e914ab14b26b2b839bbd0dc38b4]), partition=range: [(), (int32 key=1073741823)) I0628 23:56:06.352114 3366 tablet_service.cc:612] Processing CreateTablet for tablet df3659ee08ea4a5fa181023ae31550b7 (table=test-workload [id=12ac1e914ab14b26b2b839bbd0dc38b4]), partition=range: [(int32 key=1073741823), ()) I0628 23:56:06.362632 3250 ts_tablet_manager.cc:760] Registered tablet 8f3895bbb21340afbb5b7d734de4db9d I0628 23:56:06.365988 3249 ts_tablet_manager.cc:760] Registered tablet df3659ee08ea4a5fa181023ae31550b7 I0628 23:56:06.367727 3367 ts_tablet_manager.cc:760] Registered tablet 8f3895bbb21340afbb5b7d734de4db9d I0628 23:56:06.368052 3131 ts_tablet_manager.cc:760] Registered tablet df3659ee08ea4a5fa181023ae31550b7 I0628 23:56:06.368192 3444 ts_tablet_manager.cc:628] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc: Bootstrapping tablet I0628 23:56:06.369843 3366 ts_tablet_manager.cc:760] Registered tablet df3659ee08ea4a5fa181023ae31550b7 I0628 23:56:06.370899 3444 tablet_bootstrap.cc:393] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc: Bootstrap starting. I0628 23:56:06.373127 3445 ts_tablet_manager.cc:628] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0: Bootstrapping tablet I0628 23:56:06.374691 3444 tablet_bootstrap.cc:553] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:06.375782 3444 tablet_bootstrap.cc:493] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc: No blocks or log segments found. Creating new log. I0628 23:56:06.377655 3444 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:06.377764 3445 tablet_bootstrap.cc:393] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0: Bootstrap starting. I0628 23:56:06.380214 3132 ts_tablet_manager.cc:760] Registered tablet 8f3895bbb21340afbb5b7d734de4db9d I0628 23:56:06.380823 3446 ts_tablet_manager.cc:628] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Bootstrapping tablet I0628 23:56:06.381464 3445 tablet_bootstrap.cc:553] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:06.382019 3445 tablet_bootstrap.cc:493] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0: No blocks or log segments found. Creating new log. I0628 23:56:06.383826 3445 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:06.384327 3446 tablet_bootstrap.cc:393] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Bootstrap starting. I0628 23:56:06.387908 3446 tablet_bootstrap.cc:553] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:06.388391 3446 tablet_bootstrap.cc:493] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: No blocks or log segments found. Creating new log. I0628 23:56:06.390019 3444 tablet_bootstrap.cc:393] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc: No bootstrap required, opened a new log I0628 23:56:06.390390 3444 ts_tablet_manager.cc:633] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.008s I0628 23:56:06.392189 3445 tablet_bootstrap.cc:393] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0: No bootstrap required, opened a new log I0628 23:56:06.392524 3445 ts_tablet_manager.cc:633] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s I0628 23:56:06.396070 3446 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:06.398331 3445 raft_consensus.cc:269] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.398741 3444 raft_consensus.cc:269] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.399888 3445 raft_consensus.cc:301] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:06.400218 3444 raft_consensus.cc:301] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:06.400151 3445 raft_consensus.cc:499] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a618b6bf22d84b37b5540eb843f950e0, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:06.400444 3444 raft_consensus.cc:499] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12be474fbcef4f68b20a52be2d4796cc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:06.400557 3445 consensus_queue.cc:162] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:06.400848 3444 consensus_queue.cc:162] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:06.405817 3445 ts_tablet_manager.cc:659] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s I0628 23:56:06.406256 3445 ts_tablet_manager.cc:628] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0: Bootstrapping tablet I0628 23:56:06.407994 3445 tablet_bootstrap.cc:393] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0: Bootstrap starting. I0628 23:56:06.409363 3446 tablet_bootstrap.cc:393] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: No bootstrap required, opened a new log I0628 23:56:06.409657 3446 ts_tablet_manager.cc:633] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.004s I0628 23:56:06.413441 3445 tablet_bootstrap.cc:553] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:06.413846 3445 tablet_bootstrap.cc:493] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0: No blocks or log segments found. Creating new log. I0628 23:56:06.415129 3446 raft_consensus.cc:269] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.416498 3446 raft_consensus.cc:301] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:06.416697 3446 raft_consensus.cc:499] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0c1f965f1d485dbf65b77be7285067, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:06.417101 3446 consensus_queue.cc:162] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:06.418344 3445 tablet_bootstrap.cc:393] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0: No bootstrap required, opened a new log I0628 23:56:06.418560 3445 ts_tablet_manager.cc:633] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s I0628 23:56:06.423341 3445 raft_consensus.cc:269] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.424114 3445 raft_consensus.cc:301] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:06.424320 3445 raft_consensus.cc:499] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a618b6bf22d84b37b5540eb843f950e0, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:06.424686 3445 consensus_queue.cc:162] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:06.424942 3444 ts_tablet_manager.cc:659] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc: Time spent starting tablet: real 0.034s user 0.012s sys 0.020s I0628 23:56:06.425364 3444 ts_tablet_manager.cc:628] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc: Bootstrapping tablet I0628 23:56:06.427330 3444 tablet_bootstrap.cc:393] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc: Bootstrap starting. I0628 23:56:06.427839 3446 ts_tablet_manager.cc:659] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Time spent starting tablet: real 0.018s user 0.008s sys 0.004s I0628 23:56:06.428356 3446 ts_tablet_manager.cc:628] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Bootstrapping tablet I0628 23:56:06.428443 3445 ts_tablet_manager.cc:659] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s I0628 23:56:06.430044 3444 tablet_bootstrap.cc:553] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0628 23:56:06.430461 3444 tablet_bootstrap.cc:493] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc: No blocks or log segments found. Creating new log. I0628 23:56:06.430753 3446 tablet_bootstrap.cc:393] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Bootstrap starting. I0628 23:56:06.433429 3446 tablet_bootstrap.cc:553] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:06.433789 3446 tablet_bootstrap.cc:493] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: No blocks or log segments found. Creating new log. I0628 23:56:06.442368 3446 tablet_bootstrap.cc:393] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: No bootstrap required, opened a new log I0628 23:56:06.442605 3446 ts_tablet_manager.cc:633] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s I0628 23:56:06.449183 3446 raft_consensus.cc:269] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.449954 3446 raft_consensus.cc:301] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:06.450099 3446 raft_consensus.cc:499] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0c1f965f1d485dbf65b77be7285067, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:06.450456 3446 consensus_queue.cc:162] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:06.451758 3444 tablet_bootstrap.cc:393] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc: No bootstrap required, opened a new log I0628 23:56:06.451987 3444 ts_tablet_manager.cc:633] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.024s I0628 23:56:06.456575 3444 raft_consensus.cc:269] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.457330 3444 raft_consensus.cc:301] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:06.457504 3446 ts_tablet_manager.cc:659] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Time spent starting tablet: real 0.015s user 0.004s sys 0.008s I0628 23:56:06.457487 3444 raft_consensus.cc:499] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12be474fbcef4f68b20a52be2d4796cc, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:06.457811 3444 consensus_queue.cc:162] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:06.460038 3444 ts_tablet_manager.cc:659] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s I0628 23:56:06.532543 3452 raft_consensus.cc:370] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:56:06.532768 3452 raft_consensus.cc:2019] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:06.535099 3452 raft_consensus.cc:1970] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.576s I0628 23:56:06.535312 3452 raft_consensus.cc:386] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.538849 3452 leader_election.cc:223] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [CANDIDATE]: Term 1 election: Requesting vote from peer 2b0c1f965f1d485dbf65b77be7285067 I0628 23:56:06.539362 3452 leader_election.cc:223] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [CANDIDATE]: Term 1 election: Requesting vote from peer a618b6bf22d84b37b5540eb843f950e0 I0628 23:56:06.543637 3152 raft_consensus.cc:2019] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:06.545145 3387 raft_consensus.cc:2019] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:06.545708 3152 raft_consensus.cc:1970] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.604s I0628 23:56:06.547381 3152 raft_consensus.cc:1677] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12be474fbcef4f68b20a52be2d4796cc in term 1. I0628 23:56:06.547513 3387 raft_consensus.cc:1970] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.606s I0628 23:56:06.548421 3205 leader_election.cc:347] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [CANDIDATE]: Term 1 election: Vote granted by peer 2b0c1f965f1d485dbf65b77be7285067 I0628 23:56:06.548614 3205 leader_election.cc:248] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:56:06.549072 3387 raft_consensus.cc:1677] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12be474fbcef4f68b20a52be2d4796cc in term 1. I0628 23:56:06.549129 3456 raft_consensus.cc:1970] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.521s I0628 23:56:06.549366 3456 raft_consensus.cc:1859] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:56:06.549988 3204 leader_election.cc:347] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [CANDIDATE]: Term 1 election: Vote granted by peer a618b6bf22d84b37b5540eb843f950e0 I0628 23:56:06.550958 3456 raft_consensus.cc:464] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 1 LEADER]: Becoming Leader. State: Replica: 12be474fbcef4f68b20a52be2d4796cc, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:06.551621 3456 consensus_queue.cc:145] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.575532 3046 catalog_manager.cc:1673] Tablet: df3659ee08ea4a5fa181023ae31550b7 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "12be474fbcef4f68b20a52be2d4796cc" config { opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } } W0628 23:56:06.579610 3476 raft_consensus_state.cc:541] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:56:06.793156 3454 raft_consensus.cc:370] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:56:06.793324 3454 raft_consensus.cc:2019] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:06.795296 3454 raft_consensus.cc:1970] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s I0628 23:56:06.795454 3454 raft_consensus.cc:386] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.798398 3454 leader_election.cc:223] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [CANDIDATE]: Term 1 election: Requesting vote from peer 12be474fbcef4f68b20a52be2d4796cc I0628 23:56:06.798835 3454 leader_election.cc:223] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [CANDIDATE]: Term 1 election: Requesting vote from peer 2b0c1f965f1d485dbf65b77be7285067 I0628 23:56:06.803771 3152 raft_consensus.cc:2019] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:06.803916 3270 raft_consensus.cc:2019] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:06.805569 3152 raft_consensus.cc:1970] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.506s I0628 23:56:06.805841 3270 raft_consensus.cc:1970] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.537s I0628 23:56:06.807145 3152 raft_consensus.cc:1677] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a618b6bf22d84b37b5540eb843f950e0 in term 1. I0628 23:56:06.807454 3270 raft_consensus.cc:1677] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a618b6bf22d84b37b5540eb843f950e0 in term 1. I0628 23:56:06.807806 3322 leader_election.cc:347] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [CANDIDATE]: Term 1 election: Vote granted by peer 2b0c1f965f1d485dbf65b77be7285067 I0628 23:56:06.808161 3322 leader_election.cc:248] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:56:06.808425 3324 leader_election.cc:347] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [CANDIDATE]: Term 1 election: Vote granted by peer 12be474fbcef4f68b20a52be2d4796cc I0628 23:56:06.808687 3455 raft_consensus.cc:1970] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.532s I0628 23:56:06.808895 3455 raft_consensus.cc:1859] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:56:06.810215 3455 raft_consensus.cc:464] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 1 LEADER]: Becoming Leader. State: Replica: a618b6bf22d84b37b5540eb843f950e0, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:06.810948 3455 consensus_queue.cc:145] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } I0628 23:56:06.835366 3046 catalog_manager.cc:1673] Tablet: 8f3895bbb21340afbb5b7d734de4db9d reported consensus state change. New consensus state: current_term: 1 leader_uuid: "a618b6bf22d84b37b5540eb843f950e0" config { opid_index: -1 peers { permanent_uuid: "12be474fbcef4f68b20a52be2d4796cc" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 56466 } } peers { permanent_uuid: "2b0c1f965f1d485dbf65b77be7285067" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 33736 } } peers { permanent_uuid: "a618b6bf22d84b37b5540eb843f950e0" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 53218 } } } W0628 23:56:06.836280 3481 raft_consensus_state.cc:541] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:56:06.922477 27213 test_workload.cc:206] TestWorkload: Skipping table creation because table test-workload already exists I0628 23:56:07.012699 3387 raft_consensus.cc:869] T df3659ee08ea4a5fa181023ae31550b7 P a618b6bf22d84b37b5540eb843f950e0 [term 1 FOLLOWER]: Refusing update from remote peer 12be474fbcef4f68b20a52be2d4796cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:56:07.012850 3152 raft_consensus.cc:869] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Refusing update from remote peer 12be474fbcef4f68b20a52be2d4796cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:56:07.014315 3456 consensus_queue.cc:578] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [LEADER]: Connected to new peer: Peer: a618b6bf22d84b37b5540eb843f950e0, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:56:07.014714 3473 consensus_queue.cc:578] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [LEADER]: Connected to new peer: Peer: 2b0c1f965f1d485dbf65b77be7285067, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:56:07.016338 3152 raft_consensus.cc:869] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Refusing update from remote peer a618b6bf22d84b37b5540eb843f950e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:56:07.017487 3455 consensus_queue.cc:578] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [LEADER]: Connected to new peer: Peer: 2b0c1f965f1d485dbf65b77be7285067, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:56:07.024751 3270 raft_consensus.cc:869] T 8f3895bbb21340afbb5b7d734de4db9d P 12be474fbcef4f68b20a52be2d4796cc [term 1 FOLLOWER]: Refusing update from remote peer a618b6bf22d84b37b5540eb843f950e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:56:07.025914 3455 consensus_queue.cc:578] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [LEADER]: Connected to new peer: Peer: 12be474fbcef4f68b20a52be2d4796cc, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:56:07.044579 3476 raft_consensus_state.cc:534] T df3659ee08ea4a5fa181023ae31550b7 P 12be474fbcef4f68b20a52be2d4796cc [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 2 I0628 23:56:07.052717 3481 raft_consensus_state.cc:534] T 8f3895bbb21340afbb5b7d734de4db9d P a618b6bf22d84b37b5540eb843f950e0 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 2 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3316==WARNING: Can't read from symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3316==WARNING: Can't read from symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3316==WARNING: Can't read from symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3316==WARNING: Can't read from symbolizer at fd 32 ==3316==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=3316) Atomic write of size 8 at 0x7d700007f010 by thread T134: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700007f011 by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700007f000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d700007f090 by thread T134: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700007f090 by thread T135: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700007f000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d700007f088 by thread T134: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700007f088 by thread T135: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700007f000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Atomic write of size 4 at 0x7d700007f0f0 by thread T134: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700007f0f0 by thread T135: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700007f000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d700007f028 by thread T133: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700007f029 by thread T136: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadd1) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700007f000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d700007f058 by thread T133: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700007f058 by thread T135: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700007f000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d700007f050 by thread T133: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700007f052 by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700007f000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) ==3071==WARNING: external symbolizer didn't start up correctly! #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) ==3071==WARNING: Can't write to symbolizer at fd 34 #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d700007f048 by thread T133: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700007f04a by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700007f000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 1 at 0x7d700005f019 by thread T135: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700005f018 by thread T133: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700005f000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3071==WARNING: Can't read from symbolizer at fd 31 ================== WARNING: ThreadSanitizer: data race (pid=3316) Read of size 8 at 0x7d500004fd38 by thread T135: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500004fd38 by thread T136: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3071==WARNING: Can't read from symbolizer at fd 31 #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=3445, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d500004f538 by thread T134: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500004f538 by thread T136: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004f400 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=3445, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d500004f538 by thread T136: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500004f538 by thread T134: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) ==3071==WARNING: Can't read from symbolizer at fd 31 #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004f400 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=3445, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Read of size 8 at 0x7d500004fd38 by thread T133: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) ==3071==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=3071) Atomic write of size 4 at 0x7d700008f0f0 by thread T133: #0 (kudu-tserver+0x000000466be0) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700008f0f0 by thread T132: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7d700008f028 by thread T131: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700008f029 by thread T132: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #20 (libkudu_util.so+0x000000224fcf) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #21 (libkudu_util.so+0x00000022941e) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #22 (libkudu_util.so+0x00000022937b) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #23 (libkudu_util.so+0x000000229303) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #24 (libkudu_util.so+0x000000229101) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #25 (libkrpc.so+0x0000000cce71) #22 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #27 (kudu-tserver+0x000000421829) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) Previous write of size 8 at 0x7d500004fd38 by thread T136: #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== #0 (libtablet.so+0x0000001fb20c) ================== WARNING: ThreadSanitizer: data race (pid=3071) Atomic write of size 8 at 0x7d700008f018 by thread T131: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700008f019 by thread T132: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7d700008f090 by thread T131: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700008f090 by thread T132: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #1 (libtablet.so+0x0000001fad02) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #2 (libtablet.so+0x000000295cf2) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== #3 (libtablet.so+0x00000028c6b7) ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7d700008f050 by thread T130: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700008f050 by thread T131: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=3509, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7d700008f0c0 by thread T130: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700008f0c0 by thread T131: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=3509, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7d500004f538 by thread T130: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #10 (libkrpc.so+0x0000000cce71) #2 (libtablet.so+0x000000295cf2) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #3 (libtablet.so+0x00000028c6b7) #15 (libkudu_util.so+0x000000229303) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #16 (libkudu_util.so+0x000000229101) #6 (libtablet.so+0x0000001dde5d) #17 (libkrpc.so+0x0000000cce71) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #18 (libkudu_util.so+0x00000021b55f) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #19 (kudu-tserver+0x000000421829) #11 (libtablet.so+0x00000023d255) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #12 (libtablet.so+0x00000023d1ea) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #13 (libconsensus.so+0x0000000befdd) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #15 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #16 (libconsensus.so+0x0000000c5643) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 #17 (libconsensus.so+0x0000000c5401) 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #18 (libkrpc.so+0x0000000cce71) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #19 (libkudu_util.so+0x000000226bcd) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #20 (libkudu_util.so+0x000000224fcf) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #21 (libkudu_util.so+0x00000022941e) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=3445, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #22 (libkudu_util.so+0x00000022937b) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #23 (libkudu_util.so+0x000000229303) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #24 (libkudu_util.so+0x000000229101) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #25 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500004f538 by thread T131: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004f400 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=3509, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=3446, finished) created by thread T49 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3198==WARNING: Can't read from symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3198==WARNING: Can't read from symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3198==WARNING: Can't read from symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3198==WARNING: Can't read from symbolizer at fd 32 ==3198==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7da000016b60 by thread T133: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7da000016b61 by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 15936 at 0x7da000014000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Atomic write of size 8 at 0x7da00001b020 by thread T134: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7da00001b021 by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 15936 at 0x7da000018000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Atomic write of size 8 at 0x7da000016b58 by thread T133: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7da000016b59 by thread T135: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 15936 at 0x7da000014000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Atomic write of size 4 at 0x7da00001b0e0 by thread T134: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7da00001b0e0 by thread T135: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 15936 at 0x7da000018000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7da000016bd0 by thread T133: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7da000016bd0 by thread T135: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 15936 at 0x7da000014000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Atomic write of size 4 at 0x7d900004f5e0 by thread T136: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900004f5e0 by thread T135: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900003bac8 by thread T135: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900003bac9 by thread T137: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900003bac0 by thread T135: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003bac1 by thread T137: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7d900003baa8 by thread T132: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900003bab8 by thread T135: #0 (libtablet.so+0x00000028bc27) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003baa9 by thread T133: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900003bab8 by thread T137: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900003bb40 by thread T135: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900003bb40 by thread T137: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900003bbb0 by thread T137: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900003bbb0 by thread T135: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900003bba8 by thread T137: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900003bbaa by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900003bba0 by thread T137: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003bba1 by thread T134: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadd1) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900003bc20 by thread T137: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900003bc20 by thread T135: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Atomic write of size 4 at 0x7d900003bc60 by thread T137: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003bc60 by thread T135: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d900003bde8 by thread T135: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003bde9 by thread T133: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d900003bde0 by thread T135: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900003bde1 by thread T133: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Atomic write of size 8 at 0x7d900003bdd8 by thread T135: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900003bdd9 by thread T133: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7d900003be50 by thread T135: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900003be50 by thread T133: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3508, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Read of size 8 at 0x7d500004f538 by thread T135: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500004f538 by thread T137: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004f400 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=3444, finished) created by thread T49 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Atomic write of size 4 at 0x7d900004fa98 by thread T133: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900004fa98 by thread T132: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T132: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7dc000009728 by thread T136: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dc000009728 by thread T134: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb826) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3505, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3316) Write of size 8 at 0x7fc4d2fc39d0 by thread T133: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7fc4d2fc39d0 by thread T136: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7fc4d2fc1000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3503, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=3515, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7dd000020418 by thread T133: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd000020419 by thread T132: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7dd000020410 by thread T133: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dd000020410 by thread T132: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7dd000020490 by thread T133: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd000020490 by thread T132: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Atomic write of size 4 at 0x7dd0000204e8 by thread T133: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000297dca) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd0000204e8 by thread T132: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7d900004f5b0 by thread T134: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900004f5b0 by thread T133: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 4 at 0x7dd00001d9b0 by thread T133: #0 (libtablet.so+0x000000299980) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd00001d9b0 by thread T137: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299980) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7dd00001d988 by thread T133: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd00001d988 by thread T137: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fadab) #5 (libtablet.so+0x000000295cf2) #6 (libtablet.so+0x00000028c6b7) #7 (libtablet.so+0x000000289219) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7dd000020420 by thread T130: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7dd000020421 by thread T131: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=3509, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Atomic read of size 8 at 0x7dd00002e310 by thread T135: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7dd00002e310 by thread T133: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x000000297d2b) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x4661ac) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Read of size 8 at 0x7dd00002e390 by thread T135: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7dd00002e390 by thread T133: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x000000297d2b) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002964a5) #10 (libtablet.so+0x0000002963c3) #11 (libtablet.so+0x0000002962c4) #12 (libtablet.so+0x00000028ca72) #13 (libtablet.so+0x00000028933f) #14 (libtablet.so+0x0000001dcebd) #15 (libtablet.so+0x0000001dde5d) #16 (libtablet.so+0x0000001ddcaf) #17 (libtablet.so+0x0000002438f1) #18 (libtablet.so+0x00000023b2f5) #19 (libtablet.so+0x00000023d2c6) #20 (libtablet.so+0x00000023d255) #21 (libtablet.so+0x00000023d1ea) #22 (libconsensus.so+0x0000000befdd) #23 (libconsensus.so+0x0000000c5726) #24 (libconsensus.so+0x0000000c568e) #25 (libconsensus.so+0x0000000c5643) #26 (libconsensus.so+0x0000000c5401) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x000000226bcd) #29 (libkudu_util.so+0x000000224fcf) #30 (libkudu_util.so+0x00000022941e) #31 (libkudu_util.so+0x00000022937b) #32 (libkudu_util.so+0x000000229303) #33 (libkudu_util.so+0x000000229101) #34 (libkrpc.so+0x0000000cce71) #35 (libkudu_util.so+0x00000021b55f) #36 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb3fb) ================== ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 8 at 0x7fca2af19718 by thread T137: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7fca2af19719 by thread T134: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7fca2af03000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=3527, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3502, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== W0628 23:56:21.748355 3485 outbound_call.cc:202] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.106.77.1:56466, user_credentials={real_user=slave, eff_user=}} blocked reactor thread for 47040.1us ================== WARNING: ThreadSanitizer: data race (pid=3198) Write of size 1 at 0x7dd00001b769 by thread T134: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000297dca) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dd00001b768 by thread T135: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000000000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=3504, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=3512, running) created by thread T129 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7dc00001dd90 by thread T131: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dc00001dd91 by thread T132: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000010000 allocated by thread T133: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=3511, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000da1d1) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7f7bfe7a1c78 by thread T131: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7f7bfe7a1c7a by thread T133: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f7bfe787000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7f7bfe7a1c68 by thread T131: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7f7bfe7a1c68 by thread T133: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f7bfe787000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7f7bfe7a1ce0 by thread T131: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f7bfe7a1ce0 by thread T133: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f7bfe787000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3071) Write of size 8 at 0x7f7bfe7a1cd0 by thread T131: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f7bfe7a1cd0 by thread T133: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1019904 at 0x7f7bfe787000 allocated by thread T131: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=3510, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T133 'apply [worker]-' (tid=3513, running) created by thread T68 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== I0628 23:56:41.152308 3451 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:56:41.228936 3467 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:56:41.229086 3466 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:56:41.229882 3451 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2/wals/8f3895bbb21340afbb5b7d734de4db9d/wal-000000002 I0628 23:56:41.279171 3467 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1/wals/8f3895bbb21340afbb5b7d734de4db9d/wal-000000002 I0628 23:56:41.337929 3466 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0/wals/8f3895bbb21340afbb5b7d734de4db9d/wal-000000002 I0628 23:56:41.392981 3449 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:56:41.403383 3460 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:56:41.408313 3457 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:56:41.425427 3449 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-1/wals/df3659ee08ea4a5fa181023ae31550b7/wal-000000002 I0628 23:56:41.444228 3457 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0/wals/df3659ee08ea4a5fa181023ae31550b7/wal-000000002 I0628 23:56:41.445451 3460 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-2/wals/df3659ee08ea4a5fa181023ae31550b7/wal-000000002 I0628 23:56:41.590564 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 3027 I0628 23:56:41.619809 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 3198 I0628 23:56:41.665496 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 3316 I0628 23:56:41.711753 27213 delete_table-test.cc:1005] Tombstoning first tablet 8f3895bbb21340afbb5b7d734de4db9d... I0628 23:56:41.715080 3132 tablet_service.cc:653] Processing DeleteTablet for tablet 8f3895bbb21340afbb5b7d734de4db9d with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:41786 I0628 23:56:41.715467 3132 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 8f3895bbb21340afbb5b7d734de4db9d I0628 23:56:41.715829 3132 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(8f3895bbb21340afbb5b7d734de4db9d) I0628 23:56:41.716047 3132 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(8f3895bbb21340afbb5b7d734de4db9d) I0628 23:56:41.716256 3132 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(8f3895bbb21340afbb5b7d734de4db9d) I0628 23:56:41.716475 3132 maintenance_manager.cc:180] Unregistered op FlushMRSOp(8f3895bbb21340afbb5b7d734de4db9d) I0628 23:56:41.716651 3132 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(8f3895bbb21340afbb5b7d734de4db9d) I0628 23:56:41.716836 3132 maintenance_manager.cc:180] Unregistered op LogGCOp(8f3895bbb21340afbb5b7d734de4db9d) I0628 23:56:41.717208 3132 raft_consensus.cc:1495] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:56:41.717653 3132 raft_consensus.cc:1511] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:56:41.737536 3132 ts_tablet_manager.cc:955] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:56:41.742055 3132 ts_tablet_manager.cc:965] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Tablet deleted. Last logged OpId: 1.821 I0628 23:56:41.743191 3132 tablet_bootstrap.cc:393] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Deleted tablet blocks from disk I0628 23:56:41.744349 27213 delete_table-test.cc:1008] Waiting for first tablet to be tombstoned... W0628 23:56:41.751653 3192 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:41786 forcing change of non-runtime-safe flag fault_crash_after_blocks_deleted I0628 23:56:41.751899 3192 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:41786 changed flags via RPC: fault_crash_after_blocks_deleted from '0' to '1.0' I0628 23:56:41.752729 27213 delete_table-test.cc:1023] Tombstoning second tablet df3659ee08ea4a5fa181023ae31550b7... I0628 23:56:41.753499 3132 tablet_service.cc:653] Processing DeleteTablet for tablet df3659ee08ea4a5fa181023ae31550b7 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:41786 I0628 23:56:41.753720 3132 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: df3659ee08ea4a5fa181023ae31550b7 I0628 23:56:41.753989 3132 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(df3659ee08ea4a5fa181023ae31550b7) I0628 23:56:41.754240 3132 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(df3659ee08ea4a5fa181023ae31550b7) I0628 23:56:41.754472 3132 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(df3659ee08ea4a5fa181023ae31550b7) I0628 23:56:41.754679 3132 maintenance_manager.cc:180] Unregistered op FlushMRSOp(df3659ee08ea4a5fa181023ae31550b7) I0628 23:56:41.754837 3132 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(df3659ee08ea4a5fa181023ae31550b7) I0628 23:56:41.755092 3132 maintenance_manager.cc:180] Unregistered op LogGCOp(df3659ee08ea4a5fa181023ae31550b7) I0628 23:56:41.755303 3132 raft_consensus.cc:1495] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:56:41.755501 3132 raft_consensus.cc:1511] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067 [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:56:41.773149 3132 ts_tablet_manager.cc:955] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:56:41.777181 3132 ts_tablet_manager.cc:965] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Tablet deleted. Last logged OpId: 1.821 W0628 23:56:41.777390 3132 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:56:41.777479 3132 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:56:41.777683 3132 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_blocks_deleted *** Check failure stack trace: *** @ 0x7f7c4c4cf304 google::LogMessage::SendToLog() at ??:0 @ 0x7f7c4c4cf800 google::LogMessage::Flush() at ??:0 @ 0x7f7c4c4d3432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f7c4d19ae42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7f7c54056e6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f7c5409e3c4 kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7f7c5409f1dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7f7c5407c497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7f7c51c3aef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7f7c51c3ad0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f7c5176bd49 std::function<>::operator()() at ??:0 @ 0x7f7c5176b949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7f7c5176d69e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f7c5176f477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f7c5176f3db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f7c5176f384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f7c5176f1aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f7c51716e72 boost::function0<>::operator()() at ??:0 @ 0x7f7c4d205560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7f7c4ec44182 start_thread at ??:0 @ 0x7f7c4a7d347d clone at ??:0 @ (nil) (unknown) I0628 23:56:41.903548 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:33736 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=35234 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:58376 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0 --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:42.020179 3562 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:42.020722 3562 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:42.034564 3562 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:42.037431 3562 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 18898 I0628 23:56:42.048976 3562 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0 uuid: "2b0c1f965f1d485dbf65b77be7285067" format_stamp: "Formatted at 2016-06-28 23:56:05 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:42.083528 3562 ts_tablet_manager.cc:604] Loading metadata for tablet 8f3895bbb21340afbb5b7d734de4db9d I0628 23:56:42.088755 3562 ts_tablet_manager.cc:936] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_TOMBSTONED I0628 23:56:42.094473 3562 ts_tablet_manager.cc:955] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:56:42.100395 3562 ts_tablet_manager.cc:965] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Tablet deleted. Last logged OpId: 1.821 I0628 23:56:42.100929 3562 ts_tablet_manager.cc:760] Registered tablet 8f3895bbb21340afbb5b7d734de4db9d I0628 23:56:42.101033 3562 ts_tablet_manager.cc:604] Loading metadata for tablet df3659ee08ea4a5fa181023ae31550b7 I0628 23:56:42.103025 3562 ts_tablet_manager.cc:936] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_TOMBSTONED I0628 23:56:42.103140 3562 ts_tablet_manager.cc:955] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:56:42.106348 3562 ts_tablet_manager.cc:965] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Tablet deleted. Last logged OpId: 1.821 I0628 23:56:42.107540 3562 ts_tablet_manager.cc:760] Registered tablet df3659ee08ea4a5fa181023ae31550b7 I0628 23:56:42.108691 3562 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:42.200167 3562 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:33736 I0628 23:56:42.201541 3562 webserver.cc:122] Starting webserver on 127.106.77.0:35234 I0628 23:56:42.201629 3562 webserver.cc:133] Document root disabled I0628 23:56:42.203407 3562 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:35234/ I0628 23:56:42.204895 3562 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_0.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:56:42.207442 3562 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:42.212510 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 3562 I0628 23:56:42.212870 27213 delete_table-test.cc:1031] Waiting for second tablet to be tombstoned... W0628 23:56:42.214519 3676 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:58376: Network error: Failed to ping master at 127.0.0.1:58376: Client connection negotiation failed: client connection to 127.0.0.1:58376: connect: Connection refused (error 111) W0628 23:56:42.216665 3676 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:58376: Network error: Failed to ping master at 127.0.0.1:58376: Client connection negotiation failed: client connection to 127.0.0.1:58376: connect: Connection refused (error 111) W0628 23:56:42.218402 3676 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:58376: Network error: Failed to ping master at 127.0.0.1:58376: Client connection negotiation failed: client connection to 127.0.0.1:58376: connect: Connection refused (error 111) W0628 23:56:42.218655 3676 heartbeater.cc:283] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. W0628 23:56:42.222780 3676 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:58376: Network error: Failed to ping master at 127.0.0.1:58376: Client connection negotiation failed: client connection to 127.0.0.1:58376: connect: Connection refused (error 111) I0628 23:56:42.231000 27213 delete_table-test.cc:1045] Deleting all tablets... I0628 23:56:42.232296 3613 tablet_service.cc:653] Processing DeleteTablet for tablet df3659ee08ea4a5fa181023ae31550b7 with delete_type TABLET_DATA_DELETED from {real_user=slave, eff_user=} at 127.0.0.1:41789 I0628 23:56:42.232741 3613 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: df3659ee08ea4a5fa181023ae31550b7 I0628 23:56:42.233103 3613 ts_tablet_manager.cc:955] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:42.237821 3613 ts_tablet_manager.cc:965] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Tablet deleted. Last logged OpId: 1.821 I0628 23:56:42.238417 3613 tablet_bootstrap.cc:393] T df3659ee08ea4a5fa181023ae31550b7 P 2b0c1f965f1d485dbf65b77be7285067: Deleted tablet blocks from disk I0628 23:56:42.240644 3613 tablet_service.cc:653] Processing DeleteTablet for tablet 8f3895bbb21340afbb5b7d734de4db9d with delete_type TABLET_DATA_DELETED from {real_user=slave, eff_user=} at 127.0.0.1:41789 I0628 23:56:42.240859 3613 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 8f3895bbb21340afbb5b7d734de4db9d I0628 23:56:42.241008 3613 ts_tablet_manager.cc:955] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:56:42.244573 3613 ts_tablet_manager.cc:965] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Tablet deleted. Last logged OpId: 1.821 I0628 23:56:42.245066 3613 tablet_bootstrap.cc:393] T 8f3895bbb21340afbb5b7d734de4db9d P 2b0c1f965f1d485dbf65b77be7285067: Deleted tablet blocks from disk I0628 23:56:42.247407 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 3562 [ OK ] FaultFlags/DeleteTableTombstonedParamTest.TestTabletTombstone/0 (37406 ms) [ RUN ] FaultFlags/DeleteTableTombstonedParamTest.TestTabletTombstone/1 I0628 23:56:42.275599 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/master --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/master --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/master/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:42.390895 3682 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:42.391355 3682 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:42.404022 3682 master_main.cc:58] Initializing master server... I0628 23:56:42.406337 3682 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 18898 I0628 23:56:42.413298 3682 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/master/instance: No such file or directory (error 2) I0628 23:56:42.413525 3682 server_base.cc:165] Creating new FS layout I0628 23:56:42.426550 3682 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/master/instance: uuid: "9660a82aea584dc8b3cb33aa3ec6067f" format_stamp: "Formatted at 2016-06-28 23:56:42 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:42.439131 3682 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/master uuid: "9660a82aea584dc8b3cb33aa3ec6067f" format_stamp: "Formatted at 2016-06-28 23:56:42 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:42.472621 3682 master_main.cc:61] Starting Master server... I0628 23:56:42.506681 3682 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:51606 I0628 23:56:42.508119 3682 webserver.cc:122] Starting webserver on localhost:0 I0628 23:56:42.508304 3682 webserver.cc:133] Document root disabled I0628 23:56:42.509203 3682 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0628 23:56:42.510658 3682 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:60002/ I0628 23:56:42.512068 3682 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/master/info.pb I0628 23:56:42.513521 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master as pid 3682 I0628 23:56:42.514502 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:0 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_interface=127.106.77.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51606 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0 --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:42.537870 3725 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f: Bootstrap starting. I0628 23:56:42.541501 3725 tablet_bootstrap.cc:553] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:42.541996 3725 tablet_bootstrap.cc:493] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f: No blocks or log segments found. Creating new log. I0628 23:56:42.543615 3725 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:42.548739 3725 tablet_bootstrap.cc:393] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f: No bootstrap required, opened a new log I0628 23:56:42.554812 3725 raft_consensus.cc:269] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" member_type: VOTER } I0628 23:56:42.555814 3725 raft_consensus.cc:301] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:42.556114 3725 raft_consensus.cc:499] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9660a82aea584dc8b3cb33aa3ec6067f, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:42.556599 3725 consensus_queue.cc:162] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:42.556885 3725 raft_consensus.cc:314] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0628 23:56:42.557049 3725 raft_consensus.cc:370] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:56:42.557201 3725 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:42.558784 3725 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.586s I0628 23:56:42.559183 3725 raft_consensus.cc:386] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" member_type: VOTER } I0628 23:56:42.561156 3725 leader_election.cc:248] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:56:42.562845 3730 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.553s I0628 23:56:42.563154 3730 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:56:42.565488 3730 raft_consensus.cc:464] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 1 LEADER]: Becoming Leader. State: Replica: 9660a82aea584dc8b3cb33aa3ec6067f, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:42.566298 3730 consensus_queue.cc:145] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" member_type: VOTER } I0628 23:56:42.584851 3732 sys_catalog.cc:235] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9660a82aea584dc8b3cb33aa3ec6067f. Latest consensus state: current_term: 1 leader_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" config { opid_index: -1 peers { permanent_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" member_type: VOTER } } I0628 23:56:42.585561 3725 sys_catalog.cc:235] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" config { opid_index: -1 peers { permanent_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" member_type: VOTER } } I0628 23:56:42.586460 3725 sys_catalog.cc:238] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [sys.catalog]: This master's current role is: LEADER I0628 23:56:42.586057 3731 sys_catalog.cc:235] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" config { opid_index: -1 peers { permanent_uuid: "9660a82aea584dc8b3cb33aa3ec6067f" member_type: VOTER } } I0628 23:56:42.587137 3732 sys_catalog.cc:238] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [sys.catalog]: This master's current role is: LEADER I0628 23:56:42.587183 3731 sys_catalog.cc:238] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [sys.catalog]: This master's current role is: LEADER I0628 23:56:42.591595 3734 catalog_manager.cc:635] Loading table and tablet metadata into memory... I0628 23:56:42.593461 3725 sys_catalog.cc:311] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [sys.catalog]: configured and running, proceeding with master startup. I0628 23:56:42.596045 3735 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 9660a82aea584dc8b3cb33aa3ec6067f [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:56:42.596511 3682 master_main.cc:64] Master server successfully started. I0628 23:56:42.635221 3726 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:42.635753 3726 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:42.649441 3726 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:42.652384 3726 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 18898 I0628 23:56:42.658329 3726 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0/instance: No such file or directory (error 2) I0628 23:56:42.658529 3726 server_base.cc:165] Creating new FS layout I0628 23:56:42.667781 3726 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0/instance: uuid: "a39bbce9471e4c57b0ee333b2f942f2c" format_stamp: "Formatted at 2016-06-28 23:56:42 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:42.676554 3726 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0 uuid: "a39bbce9471e4c57b0ee333b2f942f2c" format_stamp: "Formatted at 2016-06-28 23:56:42 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:42.728731 3726 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:43.155699 3726 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:35201 I0628 23:56:43.157120 3726 webserver.cc:122] Starting webserver on 127.106.77.0:0 I0628 23:56:43.157239 3726 webserver.cc:133] Document root disabled I0628 23:56:43.159054 3726 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:51540/ I0628 23:56:43.160540 3726 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:56:43.163419 3726 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:43.167662 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 3726 I0628 23:56:43.168568 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1 --rpc_bind_addresses=127.106.77.1:0 --local_ip_for_outbound_sockets=127.106.77.1 --webserver_interface=127.106.77.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51606 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1 --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:43.178823 3850 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:51606 I0628 23:56:43.179420 3850 heartbeater.cc:319] Registering TS with master... I0628 23:56:43.179708 3850 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:43.182158 3702 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" instance_seqno: 1467158203131355 } with Master I0628 23:56:43.291105 3854 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:43.291558 3854 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:43.304738 3854 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:43.307689 3854 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 19398 I0628 23:56:43.314165 3854 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1/instance: No such file or directory (error 2) I0628 23:56:43.314362 3854 server_base.cc:165] Creating new FS layout I0628 23:56:43.322629 3854 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1/instance: uuid: "da5659d8f26244fa9e9ddc0f74dafd37" format_stamp: "Formatted at 2016-06-28 23:56:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:43.330592 3854 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1 uuid: "da5659d8f26244fa9e9ddc0f74dafd37" format_stamp: "Formatted at 2016-06-28 23:56:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:43.363927 3854 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:43.457926 3854 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.1:39154 I0628 23:56:43.459385 3854 webserver.cc:122] Starting webserver on 127.106.77.1:0 I0628 23:56:43.459529 3854 webserver.cc:133] Document root disabled I0628 23:56:43.461347 3854 webserver.cc:217] Webserver started. Bound to: http://127.106.77.1:41229/ I0628 23:56:43.462669 3854 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1/info.pb I0628 23:56:43.465450 3854 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:43.468696 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 3854 I0628 23:56:43.469924 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2 --rpc_bind_addresses=127.106.77.2:0 --local_ip_for_outbound_sockets=127.106.77.2 --webserver_interface=127.106.77.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:51606 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2 --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:56:43.486698 3968 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:51606 I0628 23:56:43.487100 3968 heartbeater.cc:319] Registering TS with master... I0628 23:56:43.487328 3968 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:43.488739 3702 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" instance_seqno: 1467158203439013 } with Master I0628 23:56:43.631400 3971 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:56:43.632009 3971 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:56:43.645345 3971 tablet_server_main.cc:54] Initializing tablet server... I0628 23:56:43.648296 3971 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 19398 I0628 23:56:43.654012 3971 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2/instance: No such file or directory (error 2) I0628 23:56:43.654202 3971 server_base.cc:165] Creating new FS layout I0628 23:56:43.662752 3971 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2/instance: uuid: "d6cee4249b2c4b86835c6302c327f3e2" format_stamp: "Formatted at 2016-06-28 23:56:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:43.671277 3971 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2 uuid: "d6cee4249b2c4b86835c6302c327f3e2" format_stamp: "Formatted at 2016-06-28 23:56:43 on dist-test-slave-dist-test-slave-cgs5" I0628 23:56:43.704490 3971 tablet_server_main.cc:57] Starting tablet server... I0628 23:56:43.823663 3971 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.2:48074 I0628 23:56:43.825125 3971 webserver.cc:122] Starting webserver on 127.106.77.2:0 I0628 23:56:43.825232 3971 webserver.cc:133] Document root disabled I0628 23:56:43.826599 3971 webserver.cc:217] Webserver started. Bound to: http://127.106.77.2:60486/ I0628 23:56:43.827889 3971 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2/info.pb I0628 23:56:43.830284 3971 tablet_server_main.cc:60] Tablet server successfully started. I0628 23:56:43.831272 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 3971 I0628 23:56:43.851292 4085 heartbeater.cc:254] Connected to a leader master server at 127.0.0.1:51606 I0628 23:56:43.851637 4085 heartbeater.cc:319] Registering TS with master... I0628 23:56:43.851915 4085 heartbeater.cc:325] Sending a full tablet report to master... I0628 23:56:43.853129 3702 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" instance_seqno: 1467158203805303 } with Master I0628 23:56:43.855950 27213 external_mini_cluster.cc:319] 3 TS(s) registered with Master I0628 23:56:43.869065 27213 delete_table-test.cc:948] Running with fault flag: fault_crash_after_wal_deleted I0628 23:56:43.871537 3702 catalog_manager.cc:772] CreateTable from {real_user=slave, eff_user=} at 127.0.0.1:50901: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\377\377\377?" } partition_schema { range_schema { columns { name: "key" } } } I0628 23:56:43.919209 3787 tablet_service.cc:612] Processing CreateTablet for tablet 3cdca585935947199322aae7bf0c9b8e (table=test-workload [id=45a0c7b3f21045799c045415dfbd1a48]), partition=range: [(), (int32 key=1073741823)) I0628 23:56:43.919205 3786 tablet_service.cc:612] Processing CreateTablet for tablet e0ae2a85c1c14bec8ce01e68c1c032a5 (table=test-workload [id=45a0c7b3f21045799c045415dfbd1a48]), partition=range: [(int32 key=1073741823), ()) I0628 23:56:43.919977 4022 tablet_service.cc:612] Processing CreateTablet for tablet 3cdca585935947199322aae7bf0c9b8e (table=test-workload [id=45a0c7b3f21045799c045415dfbd1a48]), partition=range: [(), (int32 key=1073741823)) I0628 23:56:43.928870 3904 tablet_service.cc:612] Processing CreateTablet for tablet e0ae2a85c1c14bec8ce01e68c1c032a5 (table=test-workload [id=45a0c7b3f21045799c045415dfbd1a48]), partition=range: [(int32 key=1073741823), ()) I0628 23:56:43.923725 4021 tablet_service.cc:612] Processing CreateTablet for tablet e0ae2a85c1c14bec8ce01e68c1c032a5 (table=test-workload [id=45a0c7b3f21045799c045415dfbd1a48]), partition=range: [(int32 key=1073741823), ()) I0628 23:56:43.931306 3905 tablet_service.cc:612] Processing CreateTablet for tablet 3cdca585935947199322aae7bf0c9b8e (table=test-workload [id=45a0c7b3f21045799c045415dfbd1a48]), partition=range: [(), (int32 key=1073741823)) I0628 23:56:43.940621 3787 ts_tablet_manager.cc:760] Registered tablet 3cdca585935947199322aae7bf0c9b8e I0628 23:56:43.940825 3786 ts_tablet_manager.cc:760] Registered tablet e0ae2a85c1c14bec8ce01e68c1c032a5 I0628 23:56:43.956035 4022 ts_tablet_manager.cc:760] Registered tablet 3cdca585935947199322aae7bf0c9b8e I0628 23:56:43.960593 3904 ts_tablet_manager.cc:760] Registered tablet e0ae2a85c1c14bec8ce01e68c1c032a5 I0628 23:56:43.960813 3905 ts_tablet_manager.cc:760] Registered tablet 3cdca585935947199322aae7bf0c9b8e I0628 23:56:43.964195 4021 ts_tablet_manager.cc:760] Registered tablet e0ae2a85c1c14bec8ce01e68c1c032a5 I0628 23:56:43.965059 4100 ts_tablet_manager.cc:628] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Bootstrapping tablet I0628 23:56:43.967484 4101 ts_tablet_manager.cc:628] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2: Bootstrapping tablet I0628 23:56:43.968658 4100 tablet_bootstrap.cc:393] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Bootstrap starting. I0628 23:56:43.970708 4101 tablet_bootstrap.cc:393] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2: Bootstrap starting. I0628 23:56:43.973508 4100 tablet_bootstrap.cc:553] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:43.974247 4100 tablet_bootstrap.cc:493] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: No blocks or log segments found. Creating new log. I0628 23:56:43.975318 4101 tablet_bootstrap.cc:553] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:43.975945 4101 tablet_bootstrap.cc:493] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2: No blocks or log segments found. Creating new log. I0628 23:56:43.977088 4100 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:43.978947 4102 ts_tablet_manager.cc:628] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37: Bootstrapping tablet I0628 23:56:43.979962 4101 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:43.981443 4102 tablet_bootstrap.cc:393] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37: Bootstrap starting. I0628 23:56:43.985975 4101 tablet_bootstrap.cc:393] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2: No bootstrap required, opened a new log I0628 23:56:43.986330 4101 ts_tablet_manager.cc:633] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s I0628 23:56:43.986685 4102 tablet_bootstrap.cc:553] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:43.987277 4102 tablet_bootstrap.cc:493] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37: No blocks or log segments found. Creating new log. I0628 23:56:43.989693 4102 log.cc:350] Log is configured to *not* fsync() on all Append() calls I0628 23:56:43.991883 4100 tablet_bootstrap.cc:393] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: No bootstrap required, opened a new log I0628 23:56:43.992483 4100 ts_tablet_manager.cc:633] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s I0628 23:56:43.994088 4101 raft_consensus.cc:269] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } I0628 23:56:43.996357 4101 raft_consensus.cc:301] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:43.996618 4101 raft_consensus.cc:499] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cee4249b2c4b86835c6302c327f3e2, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:43.997277 4101 consensus_queue.cc:162] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:43.997525 4102 tablet_bootstrap.cc:393] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37: No bootstrap required, opened a new log I0628 23:56:43.999346 4102 ts_tablet_manager.cc:633] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.012s I0628 23:56:44.001399 4101 ts_tablet_manager.cc:659] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s I0628 23:56:44.001966 4101 ts_tablet_manager.cc:628] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2: Bootstrapping tablet I0628 23:56:44.005300 4101 tablet_bootstrap.cc:393] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2: Bootstrap starting. I0628 23:56:44.008419 4100 raft_consensus.cc:269] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } I0628 23:56:44.009542 4101 tablet_bootstrap.cc:553] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:44.010321 4101 tablet_bootstrap.cc:493] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2: No blocks or log segments found. Creating new log. I0628 23:56:44.010563 4100 raft_consensus.cc:301] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:44.010908 4100 raft_consensus.cc:499] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39bbce9471e4c57b0ee333b2f942f2c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:44.011534 4100 consensus_queue.cc:162] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:44.023948 4102 raft_consensus.cc:269] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } I0628 23:56:44.026203 4101 tablet_bootstrap.cc:393] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2: No bootstrap required, opened a new log I0628 23:56:44.026227 4102 raft_consensus.cc:301] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:44.026463 4101 ts_tablet_manager.cc:633] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.008s I0628 23:56:44.026572 4102 raft_consensus.cc:499] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5659d8f26244fa9e9ddc0f74dafd37, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:44.027284 4102 consensus_queue.cc:162] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:44.027565 4100 ts_tablet_manager.cc:659] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Time spent starting tablet: real 0.035s user 0.016s sys 0.012s I0628 23:56:44.028393 4100 ts_tablet_manager.cc:628] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Bootstrapping tablet I0628 23:56:44.032891 4101 raft_consensus.cc:269] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } I0628 23:56:44.033831 4101 raft_consensus.cc:301] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:44.034003 4101 raft_consensus.cc:499] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cee4249b2c4b86835c6302c327f3e2, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:44.034354 4101 consensus_queue.cc:162] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:44.037432 4100 tablet_bootstrap.cc:393] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Bootstrap starting. I0628 23:56:44.039459 4102 ts_tablet_manager.cc:659] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37: Time spent starting tablet: real 0.040s user 0.016s sys 0.020s I0628 23:56:44.040017 4102 ts_tablet_manager.cc:628] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37: Bootstrapping tablet I0628 23:56:44.041582 4101 ts_tablet_manager.cc:659] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s I0628 23:56:44.043612 4102 tablet_bootstrap.cc:393] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37: Bootstrap starting. I0628 23:56:44.045568 4100 tablet_bootstrap.cc:553] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:44.046064 4100 tablet_bootstrap.cc:493] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: No blocks or log segments found. Creating new log. I0628 23:56:44.047497 4102 tablet_bootstrap.cc:553] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0628 23:56:44.047965 4102 tablet_bootstrap.cc:493] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37: No blocks or log segments found. Creating new log. I0628 23:56:44.054491 4102 tablet_bootstrap.cc:393] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37: No bootstrap required, opened a new log I0628 23:56:44.054766 4102 ts_tablet_manager.cc:633] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.004s I0628 23:56:44.054888 4100 tablet_bootstrap.cc:393] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: No bootstrap required, opened a new log I0628 23:56:44.055261 4100 ts_tablet_manager.cc:633] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.000s I0628 23:56:44.063127 4102 raft_consensus.cc:269] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } I0628 23:56:44.064343 4102 raft_consensus.cc:301] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:44.063755 4100 raft_consensus.cc:269] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } I0628 23:56:44.064782 4100 raft_consensus.cc:301] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0628 23:56:44.064633 4102 raft_consensus.cc:499] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5659d8f26244fa9e9ddc0f74dafd37, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:44.064940 4100 raft_consensus.cc:499] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39bbce9471e4c57b0ee333b2f942f2c, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:44.065177 4102 consensus_queue.cc:162] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:44.065337 4100 consensus_queue.cc:162] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0628 23:56:44.067945 4102 ts_tablet_manager.cc:659] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0628 23:56:44.069859 4100 ts_tablet_manager.cc:659] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s I0628 23:56:44.262264 4124 raft_consensus.cc:370] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:56:44.262500 4124 raft_consensus.cc:2019] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:44.264771 4124 raft_consensus.cc:1970] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.632s I0628 23:56:44.265025 4124 raft_consensus.cc:386] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } I0628 23:56:44.268373 4124 leader_election.cc:223] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [CANDIDATE]: Term 1 election: Requesting vote from peer d6cee4249b2c4b86835c6302c327f3e2 I0628 23:56:44.268854 4124 leader_election.cc:223] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [CANDIDATE]: Term 1 election: Requesting vote from peer a39bbce9471e4c57b0ee333b2f942f2c I0628 23:56:44.273813 4042 raft_consensus.cc:2019] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:44.276144 4042 raft_consensus.cc:1970] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.649s I0628 23:56:44.276618 3807 raft_consensus.cc:2019] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:44.277732 4042 raft_consensus.cc:1677] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5659d8f26244fa9e9ddc0f74dafd37 in term 1. I0628 23:56:44.278731 3860 leader_election.cc:347] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [CANDIDATE]: Term 1 election: Vote granted by peer d6cee4249b2c4b86835c6302c327f3e2 I0628 23:56:44.278795 3807 raft_consensus.cc:1970] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.601s I0628 23:56:44.278954 3860 leader_election.cc:248] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:56:44.279531 4126 raft_consensus.cc:1970] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.644s I0628 23:56:44.279826 4126 raft_consensus.cc:1859] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:56:44.281404 3807 raft_consensus.cc:1677] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da5659d8f26244fa9e9ddc0f74dafd37 in term 1. I0628 23:56:44.281519 4126 raft_consensus.cc:464] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 LEADER]: Becoming Leader. State: Replica: da5659d8f26244fa9e9ddc0f74dafd37, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:44.282409 3859 leader_election.cc:347] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [CANDIDATE]: Term 1 election: Vote granted by peer a39bbce9471e4c57b0ee333b2f942f2c I0628 23:56:44.282547 4126 consensus_queue.cc:145] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } I0628 23:56:44.307802 3700 catalog_manager.cc:1673] Tablet: 3cdca585935947199322aae7bf0c9b8e reported consensus state change. New consensus state: current_term: 1 leader_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" config { opid_index: -1 peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } } W0628 23:56:44.308871 4132 raft_consensus_state.cc:541] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:56:44.460080 4125 raft_consensus.cc:370] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0628 23:56:44.460302 4125 raft_consensus.cc:2019] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:44.462322 4125 raft_consensus.cc:1970] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.545s I0628 23:56:44.462546 4125 raft_consensus.cc:386] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } I0628 23:56:44.466038 4125 leader_election.cc:223] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [CANDIDATE]: Term 1 election: Requesting vote from peer da5659d8f26244fa9e9ddc0f74dafd37 I0628 23:56:44.466487 4125 leader_election.cc:223] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [CANDIDATE]: Term 1 election: Requesting vote from peer d6cee4249b2c4b86835c6302c327f3e2 I0628 23:56:44.472574 3925 raft_consensus.cc:2019] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:44.475500 3925 raft_consensus.cc:1970] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.558s I0628 23:56:44.477536 3925 raft_consensus.cc:1677] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39bbce9471e4c57b0ee333b2f942f2c in term 1. I0628 23:56:44.479187 3742 leader_election.cc:347] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [CANDIDATE]: Term 1 election: Vote granted by peer da5659d8f26244fa9e9ddc0f74dafd37 I0628 23:56:44.479178 4042 raft_consensus.cc:2019] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2 [term 0 FOLLOWER]: Advancing to term 1 I0628 23:56:44.479432 3742 leader_election.cc:248] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0628 23:56:44.479964 4127 raft_consensus.cc:1970] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.562s I0628 23:56:44.480211 4127 raft_consensus.cc:1859] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Leader election won for term 1 I0628 23:56:44.481884 4042 raft_consensus.cc:1970] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.601s I0628 23:56:44.484146 4042 raft_consensus.cc:1677] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39bbce9471e4c57b0ee333b2f942f2c in term 1. I0628 23:56:44.485038 3742 leader_election.cc:347] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [CANDIDATE]: Term 1 election: Vote granted by peer d6cee4249b2c4b86835c6302c327f3e2 I0628 23:56:44.488533 4127 raft_consensus.cc:464] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 LEADER]: Becoming Leader. State: Replica: a39bbce9471e4c57b0ee333b2f942f2c, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0628 23:56:44.489197 4127 consensus_queue.cc:145] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } I0628 23:56:44.526192 3700 catalog_manager.cc:1673] Tablet: e0ae2a85c1c14bec8ce01e68c1c032a5 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" config { opid_index: -1 peers { permanent_uuid: "a39bbce9471e4c57b0ee333b2f942f2c" member_type: VOTER last_known_addr { host: "127.106.77.0" port: 35201 } } peers { permanent_uuid: "da5659d8f26244fa9e9ddc0f74dafd37" member_type: VOTER last_known_addr { host: "127.106.77.1" port: 39154 } } peers { permanent_uuid: "d6cee4249b2c4b86835c6302c327f3e2" member_type: VOTER last_known_addr { host: "127.106.77.2" port: 48074 } } } W0628 23:56:44.527840 4137 raft_consensus_state.cc:541] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0628 23:56:44.648810 27213 test_workload.cc:206] TestWorkload: Skipping table creation because table test-workload already exists I0628 23:56:44.788161 4042 raft_consensus.cc:869] T 3cdca585935947199322aae7bf0c9b8e P d6cee4249b2c4b86835c6302c327f3e2 [term 1 FOLLOWER]: Refusing update from remote peer da5659d8f26244fa9e9ddc0f74dafd37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0628 23:56:44.802606 4126 consensus_queue.cc:578] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [LEADER]: Connected to new peer: Peer: d6cee4249b2c4b86835c6302c327f3e2, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:56:44.805517 3807 raft_consensus.cc:869] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Refusing update from remote peer da5659d8f26244fa9e9ddc0f74dafd37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0628 23:56:44.807184 4126 consensus_queue.cc:578] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [LEADER]: Connected to new peer: Peer: a39bbce9471e4c57b0ee333b2f942f2c, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:56:44.822223 4041 raft_consensus.cc:869] T e0ae2a85c1c14bec8ce01e68c1c032a5 P d6cee4249b2c4b86835c6302c327f3e2 [term 1 FOLLOWER]: Refusing update from remote peer a39bbce9471e4c57b0ee333b2f942f2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:56:44.823086 3925 raft_consensus.cc:869] T e0ae2a85c1c14bec8ce01e68c1c032a5 P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 FOLLOWER]: Refusing update from remote peer a39bbce9471e4c57b0ee333b2f942f2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0628 23:56:44.823784 4134 consensus_queue.cc:578] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [LEADER]: Connected to new peer: Peer: d6cee4249b2c4b86835c6302c327f3e2, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:56:44.824637 4127 consensus_queue.cc:578] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [LEADER]: Connected to new peer: Peer: da5659d8f26244fa9e9ddc0f74dafd37, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0628 23:56:44.858697 4156 raft_consensus_state.cc:534] T 3cdca585935947199322aae7bf0c9b8e P da5659d8f26244fa9e9ddc0f74dafd37 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0628 23:56:44.889905 4137 raft_consensus_state.cc:534] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 2 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3726==WARNING: Can't read from symbolizer at fd 33 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3726==WARNING: external symbolizer didn't start up correctly! ==3726==WARNING: Can't write to symbolizer at fd 36 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3726==WARNING: external symbolizer didn't start up correctly! ==3726==WARNING: Can't write to symbolizer at fd 36 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3726==WARNING: external symbolizer didn't start up correctly! ==3726==WARNING: Can't write to symbolizer at fd 36 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3726==WARNING: external symbolizer didn't start up correctly! ==3726==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 1 at 0x7d400003d3f8 by thread T134: #0 (libtablet.so+0x000000296e7c) #1 (libtablet.so+0x0000002968fc) #2 (libtablet.so+0x00000029659b) #3 (libtablet.so+0x00000028cafd) #4 (libtablet.so+0x00000028936b) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d400003d3f8 by thread T114 (mutexes: write M3468, write M8883): #0 (libtablet.so+0x0000001f8b2d) #1 (libtablet.so+0x0000001f8938) #2 (libtablet.so+0x0000001ebf10) #3 (libtablet.so+0x0000001e5407) #4 (libtablet.so+0x000000228b62) #5 (libtablet.so+0x000000281909) #6 (libtablet.so+0x000000280cc6) #7 (libtablet.so+0x0000002869b6) #8 (libtablet.so+0x00000028691a) #9 (libtablet.so+0x0000002868a3) #10 (libtablet.so+0x000000286679) #11 (libkrpc.so+0x0000000cce71) #12 (libkudu_util.so+0x00000021b55f) #13 (kudu-tserver+0x000000421829) Location is heap block of size 249 at 0x7d400003d300 allocated by thread T117: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x00000028c022) #11 (libtablet.so+0x0000002889d4) #12 (libtablet.so+0x0000001db496) #13 (libtablet.so+0x000000210662) #14 (libtablet.so+0x00000020f9b7) #15 (libtablet.so+0x00000020f54e) #16 (libtserver.so+0x00000014bc0e) #17 (libtserver.so+0x000000157d97) #18 (libtserver.so+0x000000157ce3) #19 (libtserver.so+0x000000157c63) #20 (libtserver.so+0x000000157a21) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Mutex M3468 (0x7d4400009b48) created at: #0 (kudu-tserver+0x000000434625) #1 (libkudu_util.so+0x0000001eb208) #2 (libtablet.so+0x00000028055a) #3 (libtserver.so+0x000000127713) #4 (kudu-tserver+0x0000004ad348) #5 (kudu-tserver+0x0000004ad1ee) #6 (libc.so.6+0x000000021ec4) Mutex M8883 (0x7d1c00018444) created at: #0 (kudu-tserver+0x00000046d6b8) #1 (libtserver.so+0x0000001124f7) #2 (libtserver.so+0x000000112470) #3 (libtserver.so+0x000000112439) #4 (libtserver.so+0x0000001123c8) #5 (libtablet.so+0x000000228b3c) #6 (libtablet.so+0x000000281909) #7 (libtablet.so+0x000000280cc6) #8 (libtablet.so+0x0000002869b6) #9 (libtablet.so+0x00000028691a) #10 (libtablet.so+0x0000002868a3) #11 (libtablet.so+0x000000286679) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x00000021b55f) #14 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T114 'maintenance_sch' (tid=3851, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libtablet.so+0x00000028351d) #3 (libtablet.so+0x000000280b95) #4 (libtserver.so+0x00000012824b) #5 (kudu-tserver+0x0000004ad40c) #6 (kudu-tserver+0x0000004ad1ee) #7 (libc.so.6+0x000000021ec4) Thread T117 'tablet-bootstra' (tid=4100, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x296e7c) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d500004f538 by thread T135: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500004f538 by thread T136: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004f400 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=4100, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Read of size 8 at 0x7d500004f538 by thread T134: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500004f538 by thread T136: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004f400 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=4100, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d500004f538 by thread T136: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500004f538 by thread T135: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004f400 allocated by thread T117: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T117 'tablet-bootstra' (tid=4100, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Atomic write of size 4 at 0x7d700009f8f0 by thread T134: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700009f8f0 by thread T135: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700009f800 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d700009f818 by thread T134: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700009f818 by thread T135: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700009f800 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 1 at 0x7d700009f811 by thread T134: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700009f810 by thread T136: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb826) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700009f800 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d700009f890 by thread T134: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700009f890 by thread T135: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700009f800 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d700009f888 by thread T134: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700009f888 by thread T136: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700009f800 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Read of size 8 at 0x7d60000dfdc0 by thread T135: #0 (libtablet.so+0x000000296189) #1 (libtablet.so+0x000000295f11) #2 (libtablet.so+0x000000295d63) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous atomic write of size 8 at 0x7d60000dfdc0 by thread T137: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289219) #7 (libtablet.so+0x0000001dcebd) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3854==WARNING: Can't read from symbolizer at fd 32 #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Location is heap block of size 996 at 0x7d60000dfc00 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x296189) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Atomic write of size 8 at 0x7d70000bf010 by thread T136: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d70000bf011 by thread T137: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d70000bf000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d70000bf040 by thread T134: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d70000bf041 by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d70000bf000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Atomic write of size 8 at 0x7d70000bf038 by thread T134: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d70000bf039 by thread T135: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3854==WARNING: Can't read from symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3971==WARNING: Can't read from symbolizer at fd 30 #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d70000bf000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d70000bf030 by thread T134: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d70000bf031 by thread T135: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d70000bf000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3971==WARNING: Can't read from symbolizer at fd 30 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3854==WARNING: Can't read from symbolizer at fd 32 /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3854==WARNING: Can't read from symbolizer at fd 32 ==3854==WARNING: Failed to use and restart external symbolizer! ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 1 at 0x7d700008f819 by thread T137: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x0000002979d1) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700008f818 by thread T138: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f800 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3971==WARNING: Can't read from symbolizer at fd 30 ================== WARNING: ThreadSanitizer: data race (pid=3854) Atomic write of size 4 at 0x7d700008f8f0 by thread T137: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700008f8f0 by thread T138: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f800 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Atomic write of size 8 at 0x7d700008f810 by thread T136: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700008f811 by thread T138: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f800 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ==3971==WARNING: Can't read from symbolizer at fd 30 #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 8 at 0x7d700008f888 by thread T136: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700008f888 by thread T138: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libtablet.so+0x0000001fc32a) #3 (libtablet.so+0x0000001fb0d9) #4 (libtablet.so+0x0000001fb1b7) #5 (libtablet.so+0x0000001fad02) #6 (libtablet.so+0x000000295cf2) #7 (libtablet.so+0x00000028c6b7) #8 (libtablet.so+0x000000289219) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f800 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Read of size 8 at 0x7d700008fec0 by thread T138: #0 (libtablet.so+0x000000296189) #1 (libtablet.so+0x000000295f11) #2 (libtablet.so+0x000000295d63) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous atomic write of size 8 at 0x7d700008fec0 by thread T137: #0 (kudu-tserver+0x00000046da28) #1 (libtablet.so+0x000000296149) #2 (libtablet.so+0x0000002960e0) #3 (libtablet.so+0x000000295ed9) #4 (libtablet.so+0x000000295cfd) #5 (libtablet.so+0x00000028c6b7) #6 (libtablet.so+0x000000289219) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f800 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x296189) ================== ==3971==WARNING: Failed to use and restart external symbolizer! ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Atomic write of size 8 at 0x7d700008f828 by thread T138: #0 (kudu-tserver+0x000000466ec0) WARNING: ThreadSanitizer: data race (pid=3971) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700008f829 by thread T137: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f800 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Write of size 8 at 0x7d700006f830 by thread T131 Thread T137 'apply [worker]-' (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) : #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d700006f831 by thread T130: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=4178, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7d700006f828 by thread T131: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f828 by thread T130: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb826) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=4178, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== /tmp/run_tha_testgCg15n/thirdparty/installed/bin/llvm-symbolizer: symbol lookup error: /tmp/run_tha_testgCg15n/build/dist-test-system-libs/libstdc++.so.6: undefined symbol: __tsan_init ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7d700006f820 by thread T131: ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 8 at 0x7d700008f808 by thread T137: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700008f808 by thread T138: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f800 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d700006f820 by thread T130: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=4178, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7d700006f818 by thread T131: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f819 by thread T129: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libkudu_util.so+0x00000021b498) #3 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7d700006f8a0 by thread T131: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d700006f8a0 by thread T130: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=4178, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Atomic write of size 4 at 0x7d700006f8f0 by thread T131: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f8f0 by thread T129: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #26 (libconsensus.so+0x0000000c5643) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) As if synchronized via sleep: #0 (kudu-tserver+0x0000004480c7) #1 (libtserver.so+0x0000001557a6) #2 (libkudu_util.so+0x00000021b498) #3 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #27 (libconsensus.so+0x0000000c5401) #1 (libkudu_util.so+0x0000001d048f) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #8 (libtablet.so+0x000000295539) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #9 (libtablet.so+0x0000002988dc) #36 (libkudu_util.so+0x00000021b55f) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 8 at 0x7d500004fd38 by thread T137: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) ================== WARNING: ThreadSanitizer: data race (pid=3971) Atomic write of size 8 at 0x7d700006f838 by thread T129: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #10 (libtablet.so+0x00000023d2c6) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d700006f839 by thread T131: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700006f800 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d500004fd38 by thread T138: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #3 (libtablet.so+0x00000020f9b7) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #4 (libtablet.so+0x00000020f54e) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #5 (libtserver.so+0x00000014bc0e) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #6 (libtserver.so+0x000000157d97) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) ================== WARNING: ThreadSanitizer: data race (pid=3971) Read of size 8 at 0x7d500004fd38 by thread T129: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500004fd38 by thread T132: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=4102, finished) created by thread T49 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb20c) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Read of size 8 at 0x7d700008f890 by thread T138: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d700008f890 by thread T137: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Location is heap block of size 1992 at 0x7d700008f800 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002988dc) #10 (libtablet.so+0x000000297af3) #11 (libtablet.so+0x000000296aeb) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #2 (libtablet.so+0x000000295cf2) #14 (libconsensus.so+0x0000000f0f2d) #3 (libtablet.so+0x00000028c6b7) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #4 (libtablet.so+0x000000289219) #17 (libconsensus.so+0x0000000d73d0) #5 (libtablet.so+0x0000001dcebd) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' #6 (libtablet.so+0x0000001dde5d) (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #7 (libtablet.so+0x0000001ddcaf) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb3fb) ================== #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) ================== WARNING: ThreadSanitizer: data race (pid=3854) Read of size 8 at 0x7d500004fd38 by thread T114 (mutexes: write M3468, write M8820): #0 (libtablet.so+0x0000001f890d) #1 (libtablet.so+0x0000001ebf10) #2 (libtablet.so+0x0000001e5407) #3 (libtablet.so+0x000000228b62) #4 (libtablet.so+0x000000281909) #5 (libtablet.so+0x000000280cc6) #6 (libtablet.so+0x0000002869b6) #7 (libtablet.so+0x00000028691a) #8 (libtablet.so+0x0000002868a3) #9 (libtablet.so+0x000000286679) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x00000021b55f) #12 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500004fd38 by thread T137: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #19 (libkudu_util.so+0x000000226bcd) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #20 (libkudu_util.so+0x000000224fcf) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #21 (libkudu_util.so+0x00000022941e) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #22 (libkudu_util.so+0x00000022937b) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=4181, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=4101, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== #23 (libkudu_util.so+0x000000229303) ================== WARNING: ThreadSanitizer: data race (pid=3971) Read of size 8 at 0x7d500004fd38 by thread T131: #0 (libtablet.so+0x0000001fb19b) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Previous write of size 8 at 0x7d500004fd38 by thread T132: #0 (libtablet.so+0x0000001fb20c) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #24 (libkudu_util.so+0x000000229101) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=4181, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=4101, finished) created by thread T50 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #25 (libkrpc.so+0x0000000cce71) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1fb19b) ================== #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 456 at 0x7d500004fc00 allocated by thread T116: #0 (kudu-tserver+0x0000004aaf73) #1 (libtablet.so+0x0000001db43e) #2 (libtablet.so+0x000000210662) #3 (libtablet.so+0x00000020f9b7) #4 (libtablet.so+0x00000020f54e) #5 (libtserver.so+0x00000014bc0e) #6 (libtserver.so+0x000000157d97) #7 (libtserver.so+0x000000157ce3) #8 (libtserver.so+0x000000157c63) #9 (libtserver.so+0x000000157a21) #10 (libkrpc.so+0x0000000cce71) #11 (libkudu_util.so+0x000000226bcd) #12 (libkudu_util.so+0x000000224fcf) #13 (libkudu_util.so+0x00000022941e) #14 (libkudu_util.so+0x00000022937b) #15 (libkudu_util.so+0x000000229303) #16 (libkudu_util.so+0x000000229101) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) Mutex M3468 (0x7d4400009b48) created at: #0 (kudu-tserver+0x000000434625) #1 (libkudu_util.so+0x0000001eb208) #2 (libtablet.so+0x00000028055a) #3 (libtserver.so+0x000000127713) #4 (kudu-tserver+0x0000004ad348) #5 (kudu-tserver+0x0000004ad1ee) #6 (libc.so.6+0x000000021ec4) Mutex M8820 (0x7d1c000186e4) created at: #0 (kudu-tserver+0x00000046d6b8) #1 (libtserver.so+0x0000001124f7) #2 (libtserver.so+0x000000112470) #3 (libtserver.so+0x000000112439) #4 (libtserver.so+0x0000001123c8) #5 (libtablet.so+0x000000228b3c) #6 (libtablet.so+0x000000281909) #7 (libtablet.so+0x000000280cc6) #8 (libtablet.so+0x0000002869b6) #9 (libtablet.so+0x00000028691a) #10 (libtablet.so+0x0000002868a3) #11 (libtablet.so+0x000000286679) #12 (libkrpc.so+0x0000000cce71) #13 (libkudu_util.so+0x00000021b55f) #14 (kudu-tserver+0x000000421829) Thread T114 'maintenance_sch' (tid=3969, running) created by main thread at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libtablet.so+0x00000028351d) #3 (libtablet.so+0x000000280b95) #4 (libtserver.so+0x00000012824b) #5 (kudu-tserver+0x0000004ad40c) #6 (kudu-tserver+0x0000004ad1ee) #7 (libc.so.6+0x000000021ec4) Thread T137 'apply [worker]-' (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T116 'tablet-bootstra' (tid=4102, finished) created by thread T49 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libtserver.so+0x00000014cd50) #7 (libtserver.so+0x00000012b940) #8 (libtserver_admin_proto.so+0x000000028a24) #9 (libtserver_admin_proto.so+0x00000002883e) #10 (libkrpc.so+0x000000121d48) #11 (libkrpc.so+0x000000121948) #12 (libkrpc.so+0x00000012369d) #13 (libkrpc.so+0x000000125476) #14 (libkrpc.so+0x0000001253da) #15 (libkrpc.so+0x000000125383) #16 (libkrpc.so+0x0000001251a9) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x00000021b55f) #19 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x1f890d) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Atomic write of size 8 at 0x7da000017418 by thread T137: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7da000017419 by thread T134: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 15936 at 0x7da000014000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7db000054c58 by thread T132: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000296aeb) #3 (libtablet.so+0x0000002965d3) #4 (libtablet.so+0x00000028cafd) #5 (libtablet.so+0x00000028936b) #6 (libtablet.so+0x0000001dcebd) #7 (libtablet.so+0x0000001dde5d) #8 (libtablet.so+0x0000001ddcaf) #9 (libtablet.so+0x0000002438f1) #10 (libtablet.so+0x00000023b2f5) #11 (libtablet.so+0x00000023d2c6) #12 (libtablet.so+0x00000023d255) #13 (libtablet.so+0x00000023d1ea) #14 (libconsensus.so+0x0000000befdd) #15 (libconsensus.so+0x0000000c5726) #16 (libconsensus.so+0x0000000c568e) #17 (libconsensus.so+0x0000000c5643) #18 (libconsensus.so+0x0000000c5401) #19 (libkrpc.so+0x0000000cce71) #20 (libkudu_util.so+0x000000226bcd) #21 (libkudu_util.so+0x000000224fcf) #22 (libkudu_util.so+0x00000022941e) #23 (libkudu_util.so+0x00000022937b) #24 (libkudu_util.so+0x000000229303) #25 (libkudu_util.so+0x000000229101) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x00000021b55f) #28 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7db000054c58 by thread T129: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 31872 at 0x7db000050000 allocated by thread T130: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=4181, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=4178, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d900003b7c8 by thread T136: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003b7c9 by thread T134: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d900003b7c0 by thread T136: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900003b7c0 by thread T134: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d900003b7b8 by thread T136: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900003b7b9 by thread T134: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 8 at 0x7d900003b510 by thread T138: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d900003b840 by thread T136: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900003b840 by thread T134: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Atomic write of size 4 at 0x7d900003b860 by thread T136: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003b860 by thread T134: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003b511 by thread T135: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4162, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 8 at 0x7d900003b508 by thread T138: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900003b50a by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4162, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 8 at 0x7d900003b4e8 by thread T138: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7d900003b4e8 by thread T135: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) ================== WARNING: ThreadSanitizer: data race (pid=3726) Atomic write of size 8 at 0x7d900003b790 by thread T137: #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4162, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Atomic write of size 8 at 0x7d900003b4d0 by thread T138: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003b4d1 by thread T135: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4162, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003b791 by thread T135: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 8 at 0x7d900003b588 by thread T138: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900003b588 by thread T135: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4162, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Atomic write of size 4 at 0x7d900003b5a0 by thread T138: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003b5a0 by thread T135: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4162, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Atomic write of size 8 at 0x7d900004f7e0 by thread T134: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7d900004f7e1 by thread T135: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T137: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7d900004f858 by thread T134: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7d900004f858 by thread T135: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900004e000 allocated by thread T137: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== W0628 23:56:49.116971 3742 outbound_call.cc:202] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.106.77.2:48074, user_credentials={real_user=slave, eff_user=}} blocked reactor thread for 47359.9us ================== WARNING: ThreadSanitizer: data race (pid=3726) Atomic write of size 8 at 0x7d900003b780 by thread T135: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7d900003b781 by thread T137: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 7968 at 0x7d900003a000 allocated by thread T136: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x000000296a7a) #14 (libtablet.so+0x0000002965d3) #15 (libtablet.so+0x00000028cafd) #16 (libtablet.so+0x00000028936b) #17 (libtablet.so+0x0000001dcebd) #18 (libtablet.so+0x0000001dde5d) #19 (libtablet.so+0x0000001ddcaf) #20 (libtablet.so+0x0000002438f1) #21 (libtablet.so+0x00000023b2f5) #22 (libtablet.so+0x00000023d2c6) #23 (libtablet.so+0x00000023d255) #24 (libtablet.so+0x00000023d1ea) #25 (libconsensus.so+0x0000000befdd) #26 (libconsensus.so+0x0000000c5726) #27 (libconsensus.so+0x0000000c568e) #28 (libconsensus.so+0x0000000c5643) #29 (libconsensus.so+0x0000000c5401) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x000000226bcd) #32 (libkudu_util.so+0x000000224fcf) #33 (libkudu_util.so+0x00000022941e) #34 (libkudu_util.so+0x00000022937b) #35 (libkudu_util.so+0x000000229303) #36 (libkudu_util.so+0x000000229101) #37 (libkrpc.so+0x0000000cce71) #38 (libkudu_util.so+0x00000021b55f) #39 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== W0628 23:56:52.401708 3742 outbound_call.cc:202] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.106.77.1:39154, user_credentials={real_user=slave, eff_user=}} blocked reactor thread for 66440.1us ================== WARNING: ThreadSanitizer: data race (pid=3971) Atomic write of size 4 at 0x7efe94e8d9b8 by thread T132: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7efe94e8d9b8 by thread T131: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7efe94e83000 allocated by thread T132: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=4181, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7efe95011a68 by thread T130: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7efe95011a69 by thread T132: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7efe95002000 allocated by thread T130: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=4178, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=4181, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7efe95011a58 by thread T130: #0 (libtablet.so+0x00000028bc27) #1 (libtablet.so+0x000000298a34) #2 (libtablet.so+0x0000002979d1) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7efe95011a58 by thread T132: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7efe95002000 allocated by thread T130: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=4178, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=4181, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x28bc27) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7efe95011ad8 by thread T130: #0 (libtablet.so+0x000000298a88) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7efe95011ad8 by thread T132: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7efe95002000 allocated by thread T130: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T130 'apply [worker]-' (tid=4178, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T132 'apply [worker]-' (tid=4181, running) created by thread T70 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298a88) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Atomic write of size 8 at 0x7f9a867a8078 by thread T138: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous read of size 1 at 0x7f9a867a8079 by thread T137: #0 (libtablet.so+0x0000001fc02e) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb7ea) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 254976 at 0x7f9a8677e000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4166, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4162, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== W0628 23:57:01.027947 3742 outbound_call.cc:202] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.106.77.2:48074, user_credentials={real_user=slave, eff_user=}} blocked reactor thread for 47561us ================== WARNING: ThreadSanitizer: data race (pid=3971) Atomic write of size 8 at 0x7dc00000caf8 by thread T131: #0 (kudu-tserver+0x000000466ec0) #1 (libtablet.so+0x000000227502) #2 (libtablet.so+0x0000002972a5) #3 (libtablet.so+0x000000297240) #4 (libtablet.so+0x00000028bc4a) #5 (libtablet.so+0x000000298a34) #6 (libtablet.so+0x0000002979d1) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000297dca) #9 (libtablet.so+0x000000296aeb) #10 (libtablet.so+0x0000002965d3) #11 (libtablet.so+0x00000028cafd) #12 (libtablet.so+0x00000028936b) #13 (libtablet.so+0x0000001dcebd) #14 (libtablet.so+0x0000001dde5d) #15 (libtablet.so+0x0000001ddcaf) #16 (libtablet.so+0x0000002438f1) #17 (libtablet.so+0x00000023b2f5) #18 (libtablet.so+0x00000023d2c6) #19 (libtablet.so+0x00000023d255) #20 (libtablet.so+0x00000023d1ea) #21 (libconsensus.so+0x0000000befdd) #22 (libconsensus.so+0x0000000c5726) #23 (libconsensus.so+0x0000000c568e) #24 (libconsensus.so+0x0000000c5643) #25 (libconsensus.so+0x0000000c5401) #26 (libkrpc.so+0x0000000cce71) #27 (libkudu_util.so+0x000000226bcd) #28 (libkudu_util.so+0x000000224fcf) #29 (libkudu_util.so+0x00000022941e) #30 (libkudu_util.so+0x00000022937b) #31 (libkudu_util.so+0x000000229303) #32 (libkudu_util.so+0x000000229101) #33 (libkrpc.so+0x0000000cce71) #34 (libkudu_util.so+0x00000021b55f) #35 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dc00000caf9 by thread T129: #0 (libtablet.so+0x0000001fc002) #1 (libtablet.so+0x0000001fbf59) #2 (libtablet.so+0x0000001fbbd2) #3 (libtablet.so+0x0000001fb8e4) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466ec0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Write of size 8 at 0x7dc00000cb70 by thread T131: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dc00000cb70 by thread T129: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3971) Atomic write of size 4 at 0x7dc00000cba0 by thread T131: #0 (kudu-tserver+0x000000466be0) #1 (libtserver.so+0x0000001573d5) #2 (libtablet.so+0x000000298ad1) #3 (libtablet.so+0x0000002979d1) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000297dca) #6 (libtablet.so+0x000000296aeb) #7 (libtablet.so+0x0000002965d3) #8 (libtablet.so+0x00000028cafd) #9 (libtablet.so+0x00000028936b) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dc00000cba0 by thread T129: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 63744 at 0x7dc000000000 allocated by thread T129: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T131 'apply [worker]-' (tid=4179, running) created by thread T69 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000d93bb) #16 (libconsensus.so+0x0000000d8c4f) #17 (libtserver.so+0x00000012dafc) #18 (libconsensus_proto.so+0x00000006d374) #19 (libconsensus_proto.so+0x00000006d18e) #20 (libkrpc.so+0x000000121d48) #21 (libkrpc.so+0x000000121948) #22 (libkrpc.so+0x00000012369d) #23 (libkrpc.so+0x000000125476) #24 (libkrpc.so+0x0000001253da) #25 (libkrpc.so+0x000000125383) #26 (libkrpc.so+0x0000001251a9) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Thread T129 'apply [worker]-' (tid=4169, running) created by thread T127 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023a704) #9 (libtablet.so+0x000000239e08) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x466be0) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 4 at 0x7f99ff3aa9b8 by thread T137: #0 (libtablet.so+0x000000299980) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7f99ff3aa9b8 by thread T136: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f99ff383000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299980) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7f99ff3aa9a0 by thread T137: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000296aeb) #5 (libtablet.so+0x0000002965d3) #6 (libtablet.so+0x00000028cafd) #7 (libtablet.so+0x00000028936b) #8 (libtablet.so+0x0000001dcebd) #9 (libtablet.so+0x0000001dde5d) #10 (libtablet.so+0x0000001ddcaf) #11 (libtablet.so+0x0000002438f1) #12 (libtablet.so+0x00000023b2f5) #13 (libtablet.so+0x00000023d2c6) #14 (libtablet.so+0x00000023d255) #15 (libtablet.so+0x00000023d1ea) #16 (libconsensus.so+0x0000000befdd) #17 (libconsensus.so+0x0000000c5726) #18 (libconsensus.so+0x0000000c568e) #19 (libconsensus.so+0x0000000c5643) #20 (libconsensus.so+0x0000000c5401) #21 (libkrpc.so+0x0000000cce71) #22 (libkudu_util.so+0x000000226bcd) #23 (libkudu_util.so+0x000000224fcf) #24 (libkudu_util.so+0x00000022941e) #25 (libkudu_util.so+0x00000022937b) #26 (libkudu_util.so+0x000000229303) #27 (libkudu_util.so+0x000000229101) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x00000021b55f) #30 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f99ff3aa9a0 by thread T136: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f99ff383000 allocated by thread T134: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T134 'apply [worker]-' (tid=4164, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f00a1) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 4 at 0x7dd0000230d8 by thread T137: #0 (libtablet.so+0x000000299980) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 4 at 0x7dd0000230d8 by thread T136: #0 (libtablet.so+0x0000001fad5e) #1 (libtablet.so+0x000000295cf2) #2 (libtablet.so+0x00000028c6b7) #3 (libtablet.so+0x000000289219) #4 (libtablet.so+0x0000001dcebd) #5 (libtablet.so+0x0000001dde5d) #6 (libtablet.so+0x0000001ddcaf) #7 (libtablet.so+0x0000002438f1) #8 (libtablet.so+0x00000023b2f5) #9 (libtablet.so+0x00000023d2c6) #10 (libtablet.so+0x00000023d255) #11 (libtablet.so+0x00000023d1ea) #12 (libconsensus.so+0x0000000befdd) #13 (libconsensus.so+0x0000000c5726) #14 (libconsensus.so+0x0000000c568e) #15 (libconsensus.so+0x0000000c5643) #16 (libconsensus.so+0x0000000c5401) #17 (libkrpc.so+0x0000000cce71) #18 (libkudu_util.so+0x000000226bcd) #19 (libkudu_util.so+0x000000224fcf) #20 (libkudu_util.so+0x00000022941e) #21 (libkudu_util.so+0x00000022937b) #22 (libkudu_util.so+0x000000229303) #23 (libkudu_util.so+0x000000229101) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x00000021b55f) #26 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299980) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7dd0000230c8 by thread T137: #0 (libtablet.so+0x000000299a06) #1 (libtablet.so+0x000000298fc4) #2 (libtablet.so+0x000000297b40) #3 (libtablet.so+0x000000297dca) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000296aeb) #6 (libtablet.so+0x0000002965d3) #7 (libtablet.so+0x00000028cafd) #8 (libtablet.so+0x00000028936b) #9 (libtablet.so+0x0000001dcebd) #10 (libtablet.so+0x0000001dde5d) #11 (libtablet.so+0x0000001ddcaf) #12 (libtablet.so+0x0000002438f1) #13 (libtablet.so+0x00000023b2f5) #14 (libtablet.so+0x00000023d2c6) #15 (libtablet.so+0x00000023d255) #16 (libtablet.so+0x00000023d1ea) #17 (libconsensus.so+0x0000000befdd) #18 (libconsensus.so+0x0000000c5726) #19 (libconsensus.so+0x0000000c568e) #20 (libconsensus.so+0x0000000c5643) #21 (libconsensus.so+0x0000000c5401) #22 (libkrpc.so+0x0000000cce71) #23 (libkudu_util.so+0x000000226bcd) #24 (libkudu_util.so+0x000000224fcf) #25 (libkudu_util.so+0x00000022941e) #26 (libkudu_util.so+0x00000022937b) #27 (libkudu_util.so+0x000000229303) #28 (libkudu_util.so+0x000000229101) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x00000021b55f) #31 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd0000230c8 by thread T136: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x299a06) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 1 at 0x7dd000023029 by thread T137: #0 (kudu-tserver+0x00000044cb1b) #1 (libtablet.so+0x0000002972b6) #2 (libtablet.so+0x000000297240) #3 (libtablet.so+0x00000028bc4a) #4 (libtablet.so+0x000000298a34) #5 (libtablet.so+0x000000297d2b) #6 (libtablet.so+0x000000297dca) #7 (libtablet.so+0x000000297dca) #8 (libtablet.so+0x000000296aeb) #9 (libtablet.so+0x0000002965d3) #10 (libtablet.so+0x00000028cafd) #11 (libtablet.so+0x00000028936b) #12 (libtablet.so+0x0000001dcebd) #13 (libtablet.so+0x0000001dde5d) #14 (libtablet.so+0x0000001ddcaf) #15 (libtablet.so+0x0000002438f1) #16 (libtablet.so+0x00000023b2f5) #17 (libtablet.so+0x00000023d2c6) #18 (libtablet.so+0x00000023d255) #19 (libtablet.so+0x00000023d1ea) #20 (libconsensus.so+0x0000000befdd) #21 (libconsensus.so+0x0000000c5726) #22 (libconsensus.so+0x0000000c568e) #23 (libconsensus.so+0x0000000c5643) #24 (libconsensus.so+0x0000000c5401) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x000000226bcd) #27 (libkudu_util.so+0x000000224fcf) #28 (libkudu_util.so+0x00000022941e) #29 (libkudu_util.so+0x00000022937b) #30 (libkudu_util.so+0x000000229303) #31 (libkudu_util.so+0x000000229101) #32 (libkrpc.so+0x0000000cce71) #33 (libkudu_util.so+0x00000021b55f) #34 (kudu-tserver+0x000000421829) Previous atomic read of size 8 at 0x7dd000023028 by thread T136: #0 (kudu-tserver+0x0000004661ac) #1 (libtserver.so+0x00000013d2ee) #2 (libtablet.so+0x0000001fbd19) #3 (libtablet.so+0x0000001fb70a) #4 (libtablet.so+0x0000001fb609) #5 (libtablet.so+0x0000001fb3e7) #6 (libtablet.so+0x0000001fad7b) #7 (libtablet.so+0x000000295cf2) #8 (libtablet.so+0x00000028c6b7) #9 (libtablet.so+0x000000289219) #10 (libtablet.so+0x0000001dcebd) #11 (libtablet.so+0x0000001dde5d) #12 (libtablet.so+0x0000001ddcaf) #13 (libtablet.so+0x0000002438f1) #14 (libtablet.so+0x00000023b2f5) #15 (libtablet.so+0x00000023d2c6) #16 (libtablet.so+0x00000023d255) #17 (libtablet.so+0x00000023d1ea) #18 (libconsensus.so+0x0000000befdd) #19 (libconsensus.so+0x0000000c5726) #20 (libconsensus.so+0x0000000c568e) #21 (libconsensus.so+0x0000000c5643) #22 (libconsensus.so+0x0000000c5401) #23 (libkrpc.so+0x0000000cce71) #24 (libkudu_util.so+0x000000226bcd) #25 (libkudu_util.so+0x000000224fcf) #26 (libkudu_util.so+0x00000022941e) #27 (libkudu_util.so+0x00000022937b) #28 (libkudu_util.so+0x000000229303) #29 (libkudu_util.so+0x000000229101) #30 (libkrpc.so+0x0000000cce71) #31 (libkudu_util.so+0x00000021b55f) #32 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver+0x44cb1b) ================== ================== WARNING: ThreadSanitizer: data race (pid=3726) Write of size 8 at 0x7dd000022ff0 by thread T137: #0 (libtablet.so+0x0000002994b2) #1 (libtablet.so+0x000000299394) #2 (libtablet.so+0x000000298916) #3 (libtablet.so+0x000000297af3) #4 (libtablet.so+0x000000297dca) #5 (libtablet.so+0x000000297dca) #6 (libtablet.so+0x000000297dca) #7 (libtablet.so+0x000000296aeb) #8 (libtablet.so+0x0000002965d3) #9 (libtablet.so+0x00000028cafd) #10 (libtablet.so+0x00000028936b) #11 (libtablet.so+0x0000001dcebd) #12 (libtablet.so+0x0000001dde5d) #13 (libtablet.so+0x0000001ddcaf) #14 (libtablet.so+0x0000002438f1) #15 (libtablet.so+0x00000023b2f5) #16 (libtablet.so+0x00000023d2c6) #17 (libtablet.so+0x00000023d255) #18 (libtablet.so+0x00000023d1ea) #19 (libconsensus.so+0x0000000befdd) #20 (libconsensus.so+0x0000000c5726) #21 (libconsensus.so+0x0000000c568e) #22 (libconsensus.so+0x0000000c5643) #23 (libconsensus.so+0x0000000c5401) #24 (libkrpc.so+0x0000000cce71) #25 (libkudu_util.so+0x000000226bcd) #26 (libkudu_util.so+0x000000224fcf) #27 (libkudu_util.so+0x00000022941e) #28 (libkudu_util.so+0x00000022937b) #29 (libkudu_util.so+0x000000229303) #30 (libkudu_util.so+0x000000229101) #31 (libkrpc.so+0x0000000cce71) #32 (libkudu_util.so+0x00000021b55f) #33 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7dd000022ff0 by thread T136: #0 (libtablet.so+0x0000001fb1cf) #1 (libtablet.so+0x0000001fad02) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 127488 at 0x7dd000020000 allocated by thread T135: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x00000029748b) #10 (libtablet.so+0x00000028bd5a) #11 (libtablet.so+0x000000296ec3) #12 (libtablet.so+0x0000002968fc) #13 (libtablet.so+0x00000029659b) #14 (libtablet.so+0x00000028cafd) #15 (libtablet.so+0x00000028936b) #16 (libtablet.so+0x0000001dcebd) #17 (libtablet.so+0x0000001dde5d) #18 (libtablet.so+0x0000001ddcaf) #19 (libtablet.so+0x0000002438f1) #20 (libtablet.so+0x00000023b2f5) #21 (libtablet.so+0x00000023d2c6) #22 (libtablet.so+0x00000023d255) #23 (libtablet.so+0x00000023d1ea) #24 (libconsensus.so+0x0000000befdd) #25 (libconsensus.so+0x0000000c5726) #26 (libconsensus.so+0x0000000c568e) #27 (libconsensus.so+0x0000000c5643) #28 (libconsensus.so+0x0000000c5401) #29 (libkrpc.so+0x0000000cce71) #30 (libkudu_util.so+0x000000226bcd) #31 (libkudu_util.so+0x000000224fcf) #32 (libkudu_util.so+0x00000022941e) #33 (libkudu_util.so+0x00000022937b) #34 (libkudu_util.so+0x000000229303) #35 (libkudu_util.so+0x000000229101) #36 (libkrpc.so+0x0000000cce71) #37 (libkudu_util.so+0x00000021b55f) #38 (kudu-tserver+0x000000421829) Thread T137 'apply [worker]-' (tid=4176, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4175, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4167, running) created by thread T130 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x2994b2) ================== ================== WARNING: ThreadSanitizer: data race (pid=3854) Write of size 8 at 0x7f9a86392950 by thread T135: #0 (libtablet.so+0x000000298ab0) #1 (libtablet.so+0x0000002979d1) #2 (libtablet.so+0x000000297dca) #3 (libtablet.so+0x000000296aeb) #4 (libtablet.so+0x0000002965d3) #5 (libtablet.so+0x00000028cafd) #6 (libtablet.so+0x00000028936b) #7 (libtablet.so+0x0000001dcebd) #8 (libtablet.so+0x0000001dde5d) #9 (libtablet.so+0x0000001ddcaf) #10 (libtablet.so+0x0000002438f1) #11 (libtablet.so+0x00000023b2f5) #12 (libtablet.so+0x00000023d2c6) #13 (libtablet.so+0x00000023d255) #14 (libtablet.so+0x00000023d1ea) #15 (libconsensus.so+0x0000000befdd) #16 (libconsensus.so+0x0000000c5726) #17 (libconsensus.so+0x0000000c568e) #18 (libconsensus.so+0x0000000c5643) #19 (libconsensus.so+0x0000000c5401) #20 (libkrpc.so+0x0000000cce71) #21 (libkudu_util.so+0x000000226bcd) #22 (libkudu_util.so+0x000000224fcf) #23 (libkudu_util.so+0x00000022941e) #24 (libkudu_util.so+0x00000022937b) #25 (libkudu_util.so+0x000000229303) #26 (libkudu_util.so+0x000000229101) #27 (libkrpc.so+0x0000000cce71) #28 (libkudu_util.so+0x00000021b55f) #29 (kudu-tserver+0x000000421829) Previous read of size 8 at 0x7f9a86392950 by thread T136: #0 (libtablet.so+0x0000001fb3fb) #1 (libtablet.so+0x0000001fad7b) #2 (libtablet.so+0x000000295cf2) #3 (libtablet.so+0x00000028c6b7) #4 (libtablet.so+0x000000289219) #5 (libtablet.so+0x0000001dcebd) #6 (libtablet.so+0x0000001dde5d) #7 (libtablet.so+0x0000001ddcaf) #8 (libtablet.so+0x0000002438f1) #9 (libtablet.so+0x00000023b2f5) #10 (libtablet.so+0x00000023d2c6) #11 (libtablet.so+0x00000023d255) #12 (libtablet.so+0x00000023d1ea) #13 (libconsensus.so+0x0000000befdd) #14 (libconsensus.so+0x0000000c5726) #15 (libconsensus.so+0x0000000c568e) #16 (libconsensus.so+0x0000000c5643) #17 (libconsensus.so+0x0000000c5401) #18 (libkrpc.so+0x0000000cce71) #19 (libkudu_util.so+0x000000226bcd) #20 (libkudu_util.so+0x000000224fcf) #21 (libkudu_util.so+0x00000022941e) #22 (libkudu_util.so+0x00000022937b) #23 (libkudu_util.so+0x000000229303) #24 (libkudu_util.so+0x000000229101) #25 (libkrpc.so+0x0000000cce71) #26 (libkudu_util.so+0x00000021b55f) #27 (kudu-tserver+0x000000421829) Location is heap block of size 509952 at 0x7f9a8633b000 allocated by thread T138: #0 (kudu-tserver+0x000000440858) #1 (libkudu_util.so+0x0000001d048f) #2 (libkudu_util.so+0x0000001d039a) #3 (libkudu_util.so+0x0000001d11a9) #4 (libkudu_util.so+0x0000001d0e5f) #5 (libkudu_util.so+0x0000001cc3ab) #6 (libkudu_util.so+0x0000001cb1d4) #7 (libkudu_util.so+0x0000001cba25) #8 (libtablet.so+0x000000295539) #9 (libtablet.so+0x0000002952ef) #10 (libtablet.so+0x0000002975c0) #11 (libtablet.so+0x000000296a00) #12 (libtablet.so+0x0000002965d3) #13 (libtablet.so+0x00000028cafd) #14 (libtablet.so+0x00000028936b) #15 (libtablet.so+0x0000001dcebd) #16 (libtablet.so+0x0000001dde5d) #17 (libtablet.so+0x0000001ddcaf) #18 (libtablet.so+0x0000002438f1) #19 (libtablet.so+0x00000023b2f5) #20 (libtablet.so+0x00000023d2c6) #21 (libtablet.so+0x00000023d255) #22 (libtablet.so+0x00000023d1ea) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T135 'apply [worker]-' (tid=4162, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T136 'apply [worker]-' (tid=4163, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) Thread T138 'apply [worker]-' (tid=4168, running) created by thread T132 at: #0 (kudu-tserver+0x00000044a123) #1 (libkudu_util.so+0x00000021ade7) #2 (libkudu_util.so+0x0000002262ae) #3 (libkudu_util.so+0x000000224304) #4 (libkudu_util.so+0x000000224902) #5 (libkudu_util.so+0x000000224704) #6 (libkudu_util.so+0x000000224632) #7 (libtablet.so+0x00000023ae76) #8 (libtablet.so+0x00000023908d) #9 (libtablet.so+0x000000234615) #10 (libtablet.so+0x000000234508) #11 (libtablet.so+0x00000023444a) #12 (libmaster_rpc.so+0x000000011c31) #13 (libconsensus.so+0x000000096280) #14 (libconsensus.so+0x0000000f0f2d) #15 (libconsensus.so+0x0000000f0017) #16 (libconsensus.so+0x0000000d76b1) #17 (libconsensus.so+0x0000000d73d0) #18 (libconsensus.so+0x0000000d79cd) #19 (libconsensus.so+0x0000000a97f2) #20 (libconsensus.so+0x0000000ba5e8) #21 (libconsensus.so+0x0000000ba558) #22 (libconsensus.so+0x0000000ba4d3) #23 (libconsensus.so+0x0000000befdd) #24 (libconsensus.so+0x0000000c5726) #25 (libconsensus.so+0x0000000c568e) #26 (libconsensus.so+0x0000000c5643) #27 (libconsensus.so+0x0000000c5401) #28 (libkrpc.so+0x0000000cce71) #29 (libkudu_util.so+0x000000226bcd) #30 (libkudu_util.so+0x000000224fcf) #31 (libkudu_util.so+0x00000022941e) #32 (libkudu_util.so+0x00000022937b) #33 (libkudu_util.so+0x000000229303) #34 (libkudu_util.so+0x000000229101) #35 (libkrpc.so+0x0000000cce71) #36 (libkudu_util.so+0x00000021b55f) #37 (kudu-tserver+0x000000421829) SUMMARY: ThreadSanitizer: data race (/tmp/run_tha_testgCg15n/build/dist-test-system-libs/libtablet.so+0x298ab0) ================== W0628 23:57:13.166631 3742 outbound_call.cc:202] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.106.77.2:48074, user_credentials={real_user=slave, eff_user=}} blocked reactor thread for 76375.9us I0628 23:57:18.475489 4123 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:57:18.515887 4110 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:57:18.526214 4116 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:57:18.561491 4116 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2/wals/e0ae2a85c1c14bec8ce01e68c1c032a5/wal-000000002 I0628 23:57:18.573089 4123 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0/wals/e0ae2a85c1c14bec8ce01e68c1c032a5/wal-000000002 I0628 23:57:18.575614 4110 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1/wals/e0ae2a85c1c14bec8ce01e68c1c032a5/wal-000000002 I0628 23:57:18.730211 4122 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:57:18.749979 4122 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-1/wals/3cdca585935947199322aae7bf0c9b8e/wal-000000002 I0628 23:57:18.817587 4107 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:57:18.848233 4106 log.cc:517] Max segment size reached. Starting new segment allocation. I0628 23:57:18.880893 4107 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0/wals/3cdca585935947199322aae7bf0c9b8e/wal-000000002 I0628 23:57:18.913599 4106 log.cc:400] Rolled over to a new segment: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-2/wals/3cdca585935947199322aae7bf0c9b8e/wal-000000002 I0628 23:57:18.921593 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-master with pid 3682 I0628 23:57:18.954463 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 3854 W0628 23:57:19.026222 3850 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:51606: Network error: Failed to send heartbeat: Client connection negotiation failed: client connection to 127.0.0.1:51606: connect: Connection refused (error 111) I0628 23:57:19.032490 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 3971 I0628 23:57:19.068840 27213 delete_table-test.cc:1005] Tombstoning first tablet e0ae2a85c1c14bec8ce01e68c1c032a5... I0628 23:57:19.071743 3787 tablet_service.cc:653] Processing DeleteTablet for tablet e0ae2a85c1c14bec8ce01e68c1c032a5 with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:34853 I0628 23:57:19.072235 3787 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: e0ae2a85c1c14bec8ce01e68c1c032a5 I0628 23:57:19.072614 3787 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(e0ae2a85c1c14bec8ce01e68c1c032a5) I0628 23:57:19.072916 3787 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(e0ae2a85c1c14bec8ce01e68c1c032a5) I0628 23:57:19.073190 3787 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(e0ae2a85c1c14bec8ce01e68c1c032a5) I0628 23:57:19.073462 3787 maintenance_manager.cc:180] Unregistered op FlushMRSOp(e0ae2a85c1c14bec8ce01e68c1c032a5) I0628 23:57:19.073710 3787 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(e0ae2a85c1c14bec8ce01e68c1c032a5) I0628 23:57:19.073878 3787 maintenance_manager.cc:180] Unregistered op LogGCOp(e0ae2a85c1c14bec8ce01e68c1c032a5) I0628 23:57:19.074117 3787 raft_consensus.cc:1495] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 LEADER]: Raft consensus shutting down. I0628 23:57:19.074928 3787 consensus_peers.cc:351] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c -> Peer d6cee4249b2c4b86835c6302c327f3e2 (127.106.77.2:48074): Closing peer: d6cee4249b2c4b86835c6302c327f3e2 I0628 23:57:19.075588 3787 consensus_peers.cc:351] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c -> Peer da5659d8f26244fa9e9ddc0f74dafd37 (127.106.77.1:39154): Closing peer: da5659d8f26244fa9e9ddc0f74dafd37 I0628 23:57:19.076474 3787 raft_consensus.cc:1511] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c [term 1 LEADER]: Raft consensus is shut down! I0628 23:57:19.096709 3787 ts_tablet_manager.cc:955] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:57:19.101586 3787 ts_tablet_manager.cc:965] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Tablet deleted. Last logged OpId: 1.823 I0628 23:57:19.102849 3787 tablet_bootstrap.cc:393] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Deleted tablet blocks from disk I0628 23:57:19.104023 27213 delete_table-test.cc:1008] Waiting for first tablet to be tombstoned... W0628 23:57:19.111160 3847 generic_service.cc:67] {real_user=slave, eff_user=} at 127.0.0.1:34853 forcing change of non-runtime-safe flag fault_crash_after_wal_deleted I0628 23:57:19.111426 3847 generic_service.cc:87] {real_user=slave, eff_user=} at 127.0.0.1:34853 changed flags via RPC: fault_crash_after_wal_deleted from '0' to '1.0' I0628 23:57:19.112336 27213 delete_table-test.cc:1023] Tombstoning second tablet 3cdca585935947199322aae7bf0c9b8e... I0628 23:57:19.113116 3787 tablet_service.cc:653] Processing DeleteTablet for tablet 3cdca585935947199322aae7bf0c9b8e with delete_type TABLET_DATA_TOMBSTONED from {real_user=slave, eff_user=} at 127.0.0.1:34853 I0628 23:57:19.113328 3787 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 3cdca585935947199322aae7bf0c9b8e I0628 23:57:19.113544 3787 maintenance_manager.cc:180] Unregistered op CompactRowSetsOp(3cdca585935947199322aae7bf0c9b8e) I0628 23:57:19.113693 3787 maintenance_manager.cc:180] Unregistered op MinorDeltaCompactionOp(3cdca585935947199322aae7bf0c9b8e) I0628 23:57:19.113834 3787 maintenance_manager.cc:180] Unregistered op MajorDeltaCompactionOp(3cdca585935947199322aae7bf0c9b8e) I0628 23:57:19.113987 3787 maintenance_manager.cc:180] Unregistered op FlushMRSOp(3cdca585935947199322aae7bf0c9b8e) I0628 23:57:19.114106 3787 maintenance_manager.cc:180] Unregistered op FlushDeltaMemStoresOp(3cdca585935947199322aae7bf0c9b8e) I0628 23:57:19.114218 3787 maintenance_manager.cc:180] Unregistered op LogGCOp(3cdca585935947199322aae7bf0c9b8e) I0628 23:57:19.114356 3787 raft_consensus.cc:1495] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Raft consensus shutting down. I0628 23:57:19.114567 3787 raft_consensus.cc:1511] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c [term 1 FOLLOWER]: Raft consensus is shut down! I0628 23:57:19.133458 3787 ts_tablet_manager.cc:955] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:57:19.137550 3787 ts_tablet_manager.cc:965] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Tablet deleted. Last logged OpId: 1.823 W0628 23:57:19.138691 3787 fault_injection.cc:38] FAULT INJECTION ENABLED! W0628 23:57:19.138845 3787 fault_injection.cc:39] THIS SERVER MAY CRASH! F0628 23:57:19.139003 3787 fault_injection.cc:74] Injected fault: FLAGS_fault_crash_after_wal_deleted *** Check failure stack trace: *** @ 0x7f9a4defb304 google::LogMessage::SendToLog() at ??:0 @ 0x7f9a4defb800 google::LogMessage::Flush() at ??:0 @ 0x7f9a4deff432 google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f9a4ebc6e42 kudu::fault_injection::DoMaybeFault() at ??:0 @ 0x7f9a55a82e6e kudu::fault_injection::MaybeFault() at ??:0 @ 0x7f9a55aca42c kudu::tserver::TSTabletManager::DeleteTabletData() at ??:0 @ 0x7f9a55acb1dd kudu::tserver::TSTabletManager::DeleteTablet() at ??:0 @ 0x7f9a55aa8497 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0 @ 0x7f9a53666ef5 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0 @ 0x7f9a53666d0f std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f9a53197d49 std::function<>::operator()() at ??:0 @ 0x7f9a53197949 kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7f9a5319969e kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f9a5319b477 boost::_mfi::mf0<>::operator()() at ??:0 @ 0x7f9a5319b3db boost::_bi::list1<>::operator()<>() at ??:0 @ 0x7f9a5319b384 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f9a5319b1aa boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f9a53142e72 boost::function0<>::operator()() at ??:0 @ 0x7f9a4ec31560 kudu::Thread::SuperviseThread() at ??:0 @ 0x42182a __tsan_thread_start_func at ??:0 @ 0x7f9a50670182 start_thread at ??:0 @ 0x7f9a4c1ff47d clone at ??:0 @ (nil) (unknown) I0628 23:57:19.267536 27213 external_mini_cluster.cc:556] Running /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0 --rpc_bind_addresses=127.106.77.0:35201 --local_ip_for_outbound_sockets=127.106.77.0 --webserver_port=51540 --webserver_interface=127.106.77.0 --tserver_master_addrs=127.0.0.1:51606 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0 --log_segment_size_mb=1 --never_fsync --server_dump_info_path=/tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0628 23:57:19.383224 4220 mem_tracker.cc:138] MemTracker: hard memory limit is 11.777661 GB I0628 23:57:19.383689 4220 mem_tracker.cc:140] MemTracker: soft memory limit is 7.066597 GB I0628 23:57:19.396848 4220 tablet_server_main.cc:54] Initializing tablet server... I0628 23:57:19.399857 4220 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 37398 I0628 23:57:19.411792 4220 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0 uuid: "a39bbce9471e4c57b0ee333b2f942f2c" format_stamp: "Formatted at 2016-06-28 23:56:42 on dist-test-slave-dist-test-slave-cgs5" I0628 23:57:19.446069 4220 ts_tablet_manager.cc:604] Loading metadata for tablet e0ae2a85c1c14bec8ce01e68c1c032a5 I0628 23:57:19.451067 4220 ts_tablet_manager.cc:936] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_TOMBSTONED I0628 23:57:19.458026 4220 ts_tablet_manager.cc:955] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:57:19.464000 4220 ts_tablet_manager.cc:965] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Tablet deleted. Last logged OpId: 1.823 I0628 23:57:19.464700 4220 ts_tablet_manager.cc:760] Registered tablet e0ae2a85c1c14bec8ce01e68c1c032a5 I0628 23:57:19.464833 4220 ts_tablet_manager.cc:604] Loading metadata for tablet 3cdca585935947199322aae7bf0c9b8e I0628 23:57:19.466857 4220 ts_tablet_manager.cc:936] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_TOMBSTONED I0628 23:57:19.466977 4220 ts_tablet_manager.cc:955] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0628 23:57:19.470196 4220 ts_tablet_manager.cc:965] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Tablet deleted. Last logged OpId: 1.823 I0628 23:57:19.470496 4220 ts_tablet_manager.cc:760] Registered tablet 3cdca585935947199322aae7bf0c9b8e I0628 23:57:19.471552 4220 tablet_server_main.cc:57] Starting tablet server... I0628 23:57:19.561666 4220 rpc_server.cc:164] RPC server started. Bound to: 127.106.77.0:35201 I0628 23:57:19.562985 4220 webserver.cc:122] Starting webserver on 127.106.77.0:51540 I0628 23:57:19.563081 4220 webserver.cc:133] Document root disabled I0628 23:57:19.564476 4220 webserver.cc:217] Webserver started. Bound to: http://127.106.77.0:51540/ I0628 23:57:19.565810 4220 server_base.cc:234] Dumped server information to /tmp/kudutest-1000/delete_table-test.FaultFlags_DeleteTableTombstonedParamTest.TestTabletTombstone_1.1467158082783852-27213/minicluster-data/ts-0/info.pb I0628 23:57:19.566445 27213 external_mini_cluster.cc:592] Started /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver as pid 4220 I0628 23:57:19.566781 27213 delete_table-test.cc:1031] Waiting for second tablet to be tombstoned... I0628 23:57:19.568277 4220 tablet_server_main.cc:60] Tablet server successfully started. W0628 23:57:19.576076 4334 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:51606: Network error: Failed to ping master at 127.0.0.1:51606: Client connection negotiation failed: client connection to 127.0.0.1:51606: connect: Connection refused (error 111) W0628 23:57:19.579272 4334 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:51606: Network error: Failed to ping master at 127.0.0.1:51606: Client connection negotiation failed: client connection to 127.0.0.1:51606: connect: Connection refused (error 111) W0628 23:57:19.580912 4334 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:51606: Network error: Failed to ping master at 127.0.0.1:51606: Client connection negotiation failed: client connection to 127.0.0.1:51606: connect: Connection refused (error 111) W0628 23:57:19.581097 4334 heartbeater.cc:283] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0628 23:57:19.582921 27213 delete_table-test.cc:1045] Deleting all tablets... W0628 23:57:19.583081 4334 heartbeater.cc:402] Failed to heartbeat to 127.0.0.1:51606: Network error: Failed to ping master at 127.0.0.1:51606: Client connection negotiation failed: client connection to 127.0.0.1:51606: connect: Connection refused (error 111) I0628 23:57:19.583876 4271 tablet_service.cc:653] Processing DeleteTablet for tablet 3cdca585935947199322aae7bf0c9b8e with delete_type TABLET_DATA_DELETED from {real_user=slave, eff_user=} at 127.0.0.1:34861 I0628 23:57:19.584278 4271 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: 3cdca585935947199322aae7bf0c9b8e I0628 23:57:19.584621 4271 ts_tablet_manager.cc:955] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:57:19.588749 4271 ts_tablet_manager.cc:965] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Tablet deleted. Last logged OpId: 1.823 I0628 23:57:19.589330 4271 tablet_bootstrap.cc:393] T 3cdca585935947199322aae7bf0c9b8e P a39bbce9471e4c57b0ee333b2f942f2c: Deleted tablet blocks from disk I0628 23:57:19.591226 4271 tablet_service.cc:653] Processing DeleteTablet for tablet e0ae2a85c1c14bec8ce01e68c1c032a5 with delete_type TABLET_DATA_DELETED from {real_user=slave, eff_user=} at 127.0.0.1:34861 I0628 23:57:19.591439 4271 tablet_peer.cc:207] Initiating TabletPeer shutdown for tablet: e0ae2a85c1c14bec8ce01e68c1c032a5 I0628 23:57:19.591575 4271 ts_tablet_manager.cc:955] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Deleting tablet data with delete state TABLET_DATA_DELETED I0628 23:57:19.594882 4271 ts_tablet_manager.cc:965] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Tablet deleted. Last logged OpId: 1.823 I0628 23:57:19.595352 4271 tablet_bootstrap.cc:393] T e0ae2a85c1c14bec8ce01e68c1c032a5 P a39bbce9471e4c57b0ee333b2f942f2c: Deleted tablet blocks from disk I0628 23:57:19.597282 27213 external_mini_cluster.cc:658] Killing /tmp/run_tha_testgCg15n/build/tsan/bin/kudu-tserver with pid 4220 [ OK ] FaultFlags/DeleteTableTombstonedParamTest.TestTabletTombstone/1 (37350 ms) [----------] 2 tests from FaultFlags/DeleteTableTombstonedParamTest (74756 ms total) [----------] Global test environment tear-down [==========] 15 tests from 3 test cases ran. (156836 ms total) [ PASSED ] 14 tests. [ FAILED ] 1 test, listed below: [ FAILED ] DeleteTableTest.TestDeleteTableWithConcurrentWrites 1 FAILED TEST