Note: Google Test filter = TabletCopyITest.TestListTabletsDuringTabletCopy [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from TabletCopyITest [ RUN ] TabletCopyITest.TestListTabletsDuringTabletCopy I0812 01:41:25.256669 14243 external_mini_cluster.cc:576] Running /data/10/dinesh/kudu/build/tsan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/master-0 --fs_data_dirs=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/master-0 --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/master-0 --server_dump_info_path=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/master-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0812 01:41:25.366739 14249 mem_tracker.cc:138] MemTracker: hard memory limit is 75.496170 GB I0812 01:41:25.367177 14249 mem_tracker.cc:140] MemTracker: soft memory limit is 45.297703 GB I0812 01:41:25.381613 14249 master_main.cc:58] Initializing master server... I0812 01:41:25.383780 14249 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 576980 I0812 01:41:25.392734 14249 server_base.cc:165] Could not load existing FS layout: Not found: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/master-0/instance: No such file or directory (error 2) I0812 01:41:25.392925 14249 server_base.cc:166] Creating new FS layout I0812 01:41:25.404026 14249 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/master-0/instance: uuid: "642d4e15b2c9484890649c2dfe35965e" format_stamp: "Formatted at 2016-08-12 08:41:25 on ve0518.halxg.cloudera.com" I0812 01:41:25.413553 14249 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/master-0 uuid: "642d4e15b2c9484890649c2dfe35965e" format_stamp: "Formatted at 2016-08-12 08:41:25 on ve0518.halxg.cloudera.com" I0812 01:41:25.443599 14249 master_main.cc:61] Starting Master server... I0812 01:41:25.479240 14249 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:43698 I0812 01:41:25.480726 14249 webserver.cc:122] Starting webserver on localhost:0 I0812 01:41:25.480828 14249 webserver.cc:133] Document root disabled I0812 01:41:25.481679 14249 net_util.cc:183] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0812 01:41:25.483289 14249 webserver.cc:217] Webserver started. Bound to: http://127.0.0.1:55421/ I0812 01:41:25.484761 14249 server_base.cc:235] Dumped server information to /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/master-0/info.pb I0812 01:41:25.493492 14243 external_mini_cluster.cc:612] Started /data/10/dinesh/kudu/build/tsan/bin/kudu-master as pid 14249 I0812 01:41:25.494588 14243 external_mini_cluster.cc:576] Running /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0 --rpc_bind_addresses=127.55.163.0:0 --local_ip_for_outbound_sockets=127.55.163.0 --webserver_interface=127.55.163.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43698 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0 --enable_exactly_once --never_fsync --server_dump_info_path=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0812 01:41:25.507175 14292 tablet_bootstrap.cc:400] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e: Bootstrap starting. I0812 01:41:25.511088 14292 tablet_bootstrap.cc:563] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0812 01:41:25.511560 14292 tablet_bootstrap.cc:503] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e: No blocks or log segments found. Creating new log. I0812 01:41:25.513344 14292 log.cc:349] Log is configured to *not* fsync() on all Append() calls I0812 01:41:25.517488 14292 tablet_bootstrap.cc:400] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e: No bootstrap required, opened a new log I0812 01:41:25.523015 14292 raft_consensus.cc:269] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "642d4e15b2c9484890649c2dfe35965e" member_type: VOTER } I0812 01:41:25.523999 14292 raft_consensus.cc:301] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0812 01:41:25.524201 14292 raft_consensus.cc:499] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642d4e15b2c9484890649c2dfe35965e, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0812 01:41:25.524534 14292 consensus_queue.cc:162] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0812 01:41:25.524809 14292 raft_consensus.cc:314] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0812 01:41:25.524953 14292 raft_consensus.cc:370] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0812 01:41:25.525071 14292 raft_consensus.cc:2019] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 0 FOLLOWER]: Advancing to term 1 I0812 01:41:25.526329 14292 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.561s I0812 01:41:25.526494 14292 raft_consensus.cc:386] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "642d4e15b2c9484890649c2dfe35965e" member_type: VOTER } I0812 01:41:25.528306 14292 leader_election.cc:248] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0812 01:41:25.529634 14297 raft_consensus.cc:1970] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.502s I0812 01:41:25.529850 14297 raft_consensus.cc:1859] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 1 FOLLOWER]: Leader election won for term 1 I0812 01:41:25.531852 14297 raft_consensus.cc:464] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 1 LEADER]: Becoming Leader. State: Replica: 642d4e15b2c9484890649c2dfe35965e, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0812 01:41:25.532398 14297 consensus_queue.cc:145] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "642d4e15b2c9484890649c2dfe35965e" member_type: VOTER } I0812 01:41:25.555182 14292 sys_catalog.cc:235] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "642d4e15b2c9484890649c2dfe35965e" config { opid_index: -1 peers { permanent_uuid: "642d4e15b2c9484890649c2dfe35965e" member_type: VOTER } } I0812 01:41:25.555093 14299 sys_catalog.cc:235] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 642d4e15b2c9484890649c2dfe35965e. Latest consensus state: current_term: 1 leader_uuid: "642d4e15b2c9484890649c2dfe35965e" config { opid_index: -1 peers { permanent_uuid: "642d4e15b2c9484890649c2dfe35965e" member_type: VOTER } } I0812 01:41:25.555562 14292 sys_catalog.cc:238] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [sys.catalog]: This master's current role is: LEADER I0812 01:41:25.555268 14298 sys_catalog.cc:235] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "642d4e15b2c9484890649c2dfe35965e" config { opid_index: -1 peers { permanent_uuid: "642d4e15b2c9484890649c2dfe35965e" member_type: VOTER } } I0812 01:41:25.555613 14299 sys_catalog.cc:238] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [sys.catalog]: This master's current role is: LEADER I0812 01:41:25.555827 14298 sys_catalog.cc:238] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [sys.catalog]: This master's current role is: LEADER I0812 01:41:25.557433 14300 catalog_manager.cc:655] Loading table and tablet metadata into memory... I0812 01:41:25.558269 14301 raft_consensus_state.cc:534] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0812 01:41:25.558883 14292 sys_catalog.cc:313] T 00000000000000000000000000000000 P 642d4e15b2c9484890649c2dfe35965e [sys.catalog]: configured and running, proceeding with master startup. I0812 01:41:25.567118 14249 master_main.cc:64] Master server successfully started. I0812 01:41:25.609503 14293 mem_tracker.cc:138] MemTracker: hard memory limit is 75.496170 GB I0812 01:41:25.609948 14293 mem_tracker.cc:140] MemTracker: soft memory limit is 45.297703 GB I0812 01:41:25.624820 14293 tablet_server_main.cc:54] Initializing tablet server... I0812 01:41:25.627897 14293 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 576980 I0812 01:41:25.635664 14293 server_base.cc:165] Could not load existing FS layout: Not found: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/instance: No such file or directory (error 2) I0812 01:41:25.635848 14293 server_base.cc:166] Creating new FS layout I0812 01:41:25.645036 14293 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/instance: uuid: "01494f0d9d974c5c86f7a027ae18de0e" format_stamp: "Formatted at 2016-08-12 08:41:25 on ve0518.halxg.cloudera.com" I0812 01:41:25.651805 14293 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0 uuid: "01494f0d9d974c5c86f7a027ae18de0e" format_stamp: "Formatted at 2016-08-12 08:41:25 on ve0518.halxg.cloudera.com" I0812 01:41:25.681771 14293 tablet_server_main.cc:57] Starting tablet server... I0812 01:41:25.768719 14293 rpc_server.cc:164] RPC server started. Bound to: 127.55.163.0:49885 I0812 01:41:25.770356 14293 webserver.cc:122] Starting webserver on 127.55.163.0:0 I0812 01:41:25.770449 14293 webserver.cc:133] Document root disabled I0812 01:41:25.772001 14293 webserver.cc:217] Webserver started. Bound to: http://127.55.163.0:54775/ I0812 01:41:25.773260 14293 server_base.cc:235] Dumped server information to /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/info.pb I0812 01:41:25.775367 14293 tablet_server_main.cc:60] Tablet server successfully started. I0812 01:41:25.780467 14243 external_mini_cluster.cc:612] Started /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver as pid 14293 I0812 01:41:25.781466 14243 external_mini_cluster.cc:576] Running /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-1 --rpc_bind_addresses=127.55.163.1:0 --local_ip_for_outbound_sockets=127.55.163.1 --webserver_interface=127.55.163.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43698 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-1 --enable_exactly_once --never_fsync --server_dump_info_path=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0812 01:41:25.788319 14417 heartbeater.cc:294] Connected to a master server at 127.0.0.1:43698 I0812 01:41:25.788636 14417 heartbeater.cc:361] Registering TS with master... I0812 01:41:25.788883 14417 heartbeater.cc:375] Master 127.0.0.1:43698 requested a full tablet report, sending... I0812 01:41:25.790616 14269 ts_manager.cc:83] Registered new tserver permanent_uuid: "01494f0d9d974c5c86f7a027ae18de0e" instance_seqno: 1470991285751078 with Master I0812 01:41:25.897121 14421 mem_tracker.cc:138] MemTracker: hard memory limit is 75.496170 GB I0812 01:41:25.897572 14421 mem_tracker.cc:140] MemTracker: soft memory limit is 45.297703 GB I0812 01:41:25.913507 14421 tablet_server_main.cc:54] Initializing tablet server... I0812 01:41:25.916829 14421 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 576980 I0812 01:41:25.924991 14421 server_base.cc:165] Could not load existing FS layout: Not found: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-1/instance: No such file or directory (error 2) I0812 01:41:25.925186 14421 server_base.cc:166] Creating new FS layout I0812 01:41:25.935138 14421 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-1/instance: uuid: "2b973a1abf534e0395b188635b112e01" format_stamp: "Formatted at 2016-08-12 08:41:25 on ve0518.halxg.cloudera.com" I0812 01:41:25.942674 14421 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-1 uuid: "2b973a1abf534e0395b188635b112e01" format_stamp: "Formatted at 2016-08-12 08:41:25 on ve0518.halxg.cloudera.com" I0812 01:41:25.975302 14421 tablet_server_main.cc:57] Starting tablet server... I0812 01:41:26.070273 14421 rpc_server.cc:164] RPC server started. Bound to: 127.55.163.1:54503 I0812 01:41:26.071997 14421 webserver.cc:122] Starting webserver on 127.55.163.1:0 I0812 01:41:26.072091 14421 webserver.cc:133] Document root disabled I0812 01:41:26.073729 14421 webserver.cc:217] Webserver started. Bound to: http://127.55.163.1:49565/ I0812 01:41:26.075060 14421 server_base.cc:235] Dumped server information to /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-1/info.pb I0812 01:41:26.077220 14421 tablet_server_main.cc:60] Tablet server successfully started. I0812 01:41:26.077958 14243 external_mini_cluster.cc:612] Started /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver as pid 14421 I0812 01:41:26.078618 14243 external_mini_cluster.cc:576] Running /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-2 --rpc_bind_addresses=127.55.163.2:0 --local_ip_for_outbound_sockets=127.55.163.2 --webserver_interface=127.55.163.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:43698 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-2 --enable_exactly_once --never_fsync --server_dump_info_path=/tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0812 01:41:26.085983 14536 heartbeater.cc:294] Connected to a master server at 127.0.0.1:43698 I0812 01:41:26.086323 14536 heartbeater.cc:361] Registering TS with master... I0812 01:41:26.086586 14536 heartbeater.cc:375] Master 127.0.0.1:43698 requested a full tablet report, sending... I0812 01:41:26.087988 14269 ts_manager.cc:83] Registered new tserver permanent_uuid: "2b973a1abf534e0395b188635b112e01" instance_seqno: 1470991286051128 with Master I0812 01:41:26.186689 14539 mem_tracker.cc:138] MemTracker: hard memory limit is 75.496170 GB I0812 01:41:26.187116 14539 mem_tracker.cc:140] MemTracker: soft memory limit is 45.297703 GB I0812 01:41:26.202122 14539 tablet_server_main.cc:54] Initializing tablet server... I0812 01:41:26.205265 14539 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 577480 I0812 01:41:26.213167 14539 server_base.cc:165] Could not load existing FS layout: Not found: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-2/instance: No such file or directory (error 2) I0812 01:41:26.213342 14539 server_base.cc:166] Creating new FS layout I0812 01:41:26.222775 14539 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-2/instance: uuid: "cb1c732a162c403f9b391557aa6915c5" format_stamp: "Formatted at 2016-08-12 08:41:26 on ve0518.halxg.cloudera.com" I0812 01:41:26.230336 14539 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-2 uuid: "cb1c732a162c403f9b391557aa6915c5" format_stamp: "Formatted at 2016-08-12 08:41:26 on ve0518.halxg.cloudera.com" I0812 01:41:26.261386 14539 tablet_server_main.cc:57] Starting tablet server... I0812 01:41:26.356945 14539 rpc_server.cc:164] RPC server started. Bound to: 127.55.163.2:34073 I0812 01:41:26.358644 14539 webserver.cc:122] Starting webserver on 127.55.163.2:0 I0812 01:41:26.358764 14539 webserver.cc:133] Document root disabled I0812 01:41:26.360359 14539 webserver.cc:217] Webserver started. Bound to: http://127.55.163.2:34080/ I0812 01:41:26.361732 14539 server_base.cc:235] Dumped server information to /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-2/info.pb I0812 01:41:26.364055 14243 external_mini_cluster.cc:612] Started /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver as pid 14539 I0812 01:41:26.364121 14539 tablet_server_main.cc:60] Tablet server successfully started. I0812 01:41:26.373822 14655 heartbeater.cc:294] Connected to a master server at 127.0.0.1:43698 I0812 01:41:26.374230 14655 heartbeater.cc:361] Registering TS with master... I0812 01:41:26.374498 14655 heartbeater.cc:375] Master 127.0.0.1:43698 requested a full tablet report, sending... I0812 01:41:26.376821 14269 ts_manager.cc:83] Registered new tserver permanent_uuid: "cb1c732a162c403f9b391557aa6915c5" instance_seqno: 1470991286337879 with Master I0812 01:41:26.381762 14243 external_mini_cluster.cc:339] 3 TS(s) registered with all masters I0812 01:41:26.421169 14269 catalog_manager.cc:782] CreateTable from {real_user=dinesh, eff_user=} at 127.0.0.1:58177: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0812 01:41:26.464828 14591 tablet_service.cc:617] Processing CreateTablet for tablet 5644aac95f434b23a28a109b37c3965a (table=test-workload [id=031f1dc22d7c4b0fae6750597ac9fbc9]), partition=range: [(), ()) I0812 01:41:26.464833 14473 tablet_service.cc:617] Processing CreateTablet for tablet 5644aac95f434b23a28a109b37c3965a (table=test-workload [id=031f1dc22d7c4b0fae6750597ac9fbc9]), partition=range: [(), ()) I0812 01:41:26.465123 14354 tablet_service.cc:617] Processing CreateTablet for tablet 5644aac95f434b23a28a109b37c3965a (table=test-workload [id=031f1dc22d7c4b0fae6750597ac9fbc9]), partition=range: [(), ()) I0812 01:41:26.480059 14591 ts_tablet_manager.cc:763] Registered tablet 5644aac95f434b23a28a109b37c3965a I0812 01:41:26.480060 14473 ts_tablet_manager.cc:763] Registered tablet 5644aac95f434b23a28a109b37c3965a I0812 01:41:26.480062 14354 ts_tablet_manager.cc:763] Registered tablet 5644aac95f434b23a28a109b37c3965a I0812 01:41:26.481601 14676 ts_tablet_manager.cc:629] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01: Bootstrapping tablet I0812 01:41:26.481640 14675 ts_tablet_manager.cc:629] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Bootstrapping tablet I0812 01:41:26.481729 14677 ts_tablet_manager.cc:629] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5: Bootstrapping tablet I0812 01:41:26.483772 14676 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01: Bootstrap starting. I0812 01:41:26.483875 14675 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Bootstrap starting. I0812 01:41:26.484362 14677 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5: Bootstrap starting. I0812 01:41:26.487653 14676 tablet_bootstrap.cc:563] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0812 01:41:26.487692 14675 tablet_bootstrap.cc:563] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0812 01:41:26.488148 14677 tablet_bootstrap.cc:563] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0812 01:41:26.488193 14675 tablet_bootstrap.cc:503] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: No blocks or log segments found. Creating new log. I0812 01:41:26.488188 14676 tablet_bootstrap.cc:503] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01: No blocks or log segments found. Creating new log. I0812 01:41:26.488739 14677 tablet_bootstrap.cc:503] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5: No blocks or log segments found. Creating new log. I0812 01:41:26.490175 14675 log.cc:349] Log is configured to *not* fsync() on all Append() calls I0812 01:41:26.490281 14676 log.cc:349] Log is configured to *not* fsync() on all Append() calls I0812 01:41:26.490768 14677 log.cc:349] Log is configured to *not* fsync() on all Append() calls I0812 01:41:26.494496 14676 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01: No bootstrap required, opened a new log I0812 01:41:26.494575 14675 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: No bootstrap required, opened a new log I0812 01:41:26.494827 14676 ts_tablet_manager.cc:634] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.001s I0812 01:41:26.494922 14675 ts_tablet_manager.cc:634] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s I0812 01:41:26.495014 14677 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5: No bootstrap required, opened a new log I0812 01:41:26.495334 14677 ts_tablet_manager.cc:634] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.005s I0812 01:41:26.501230 14675 raft_consensus.cc:269] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "2b973a1abf534e0395b188635b112e01" member_type: VOTER last_known_addr { host: "127.55.163.1" port: 54503 } } peers { permanent_uuid: "cb1c732a162c403f9b391557aa6915c5" member_type: VOTER last_known_addr { host: "127.55.163.2" port: 34073 } } peers { permanent_uuid: "01494f0d9d974c5c86f7a027ae18de0e" member_type: VOTER last_known_addr { host: "127.55.163.0" port: 49885 } } I0812 01:41:26.501387 14676 raft_consensus.cc:269] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "2b973a1abf534e0395b188635b112e01" member_type: VOTER last_known_addr { host: "127.55.163.1" port: 54503 } } peers { permanent_uuid: "cb1c732a162c403f9b391557aa6915c5" member_type: VOTER last_known_addr { host: "127.55.163.2" port: 34073 } } peers { permanent_uuid: "01494f0d9d974c5c86f7a027ae18de0e" member_type: VOTER last_known_addr { host: "127.55.163.0" port: 49885 } } I0812 01:41:26.501811 14677 raft_consensus.cc:269] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "2b973a1abf534e0395b188635b112e01" member_type: VOTER last_known_addr { host: "127.55.163.1" port: 54503 } } peers { permanent_uuid: "cb1c732a162c403f9b391557aa6915c5" member_type: VOTER last_known_addr { host: "127.55.163.2" port: 34073 } } peers { permanent_uuid: "01494f0d9d974c5c86f7a027ae18de0e" member_type: VOTER last_known_addr { host: "127.55.163.0" port: 49885 } } I0812 01:41:26.502955 14675 raft_consensus.cc:301] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0812 01:41:26.503069 14676 raft_consensus.cc:301] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0812 01:41:26.503213 14675 raft_consensus.cc:499] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01494f0d9d974c5c86f7a027ae18de0e, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0812 01:41:26.503371 14677 raft_consensus.cc:301] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0812 01:41:26.503307 14676 raft_consensus.cc:499] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b973a1abf534e0395b188635b112e01, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0812 01:41:26.503645 14675 consensus_queue.cc:162] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0812 01:41:26.503617 14677 raft_consensus.cc:499] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1c732a162c403f9b391557aa6915c5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0812 01:41:26.503772 14676 consensus_queue.cc:162] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0812 01:41:26.504088 14677 consensus_queue.cc:162] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0812 01:41:26.505556 14417 heartbeater.cc:367] Master 127.0.0.1:43698 was elected leader, sending a full tablet report... I0812 01:41:26.505640 14536 heartbeater.cc:367] Master 127.0.0.1:43698 was elected leader, sending a full tablet report... I0812 01:41:26.505877 14655 heartbeater.cc:367] Master 127.0.0.1:43698 was elected leader, sending a full tablet report... I0812 01:41:26.506575 14676 ts_tablet_manager.cc:661] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s I0812 01:41:26.506574 14675 ts_tablet_manager.cc:661] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s I0812 01:41:26.506878 14677 ts_tablet_manager.cc:661] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s I0812 01:41:26.758442 14686 raft_consensus.cc:370] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0812 01:41:26.758635 14686 raft_consensus.cc:2019] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 0 FOLLOWER]: Advancing to term 1 I0812 01:41:26.760817 14686 raft_consensus.cc:1970] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.509s I0812 01:41:26.761009 14686 raft_consensus.cc:386] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 peers { permanent_uuid: "2b973a1abf534e0395b188635b112e01" member_type: VOTER last_known_addr { host: "127.55.163.1" port: 54503 } } peers { permanent_uuid: "cb1c732a162c403f9b391557aa6915c5" member_type: VOTER last_known_addr { host: "127.55.163.2" port: 34073 } } peers { permanent_uuid: "01494f0d9d974c5c86f7a027ae18de0e" member_type: VOTER last_known_addr { host: "127.55.163.0" port: 49885 } } I0812 01:41:26.764439 14686 leader_election.cc:223] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [CANDIDATE]: Term 1 election: Requesting vote from peer 2b973a1abf534e0395b188635b112e01 I0812 01:41:26.765003 14686 leader_election.cc:223] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [CANDIDATE]: Term 1 election: Requesting vote from peer 01494f0d9d974c5c86f7a027ae18de0e I0812 01:41:26.769639 14493 raft_consensus.cc:2019] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01 [term 0 FOLLOWER]: Advancing to term 1 I0812 01:41:26.771390 14374 raft_consensus.cc:2019] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 0 FOLLOWER]: Advancing to term 1 I0812 01:41:26.772116 14493 raft_consensus.cc:1970] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.634s I0812 01:41:26.773669 14374 raft_consensus.cc:1970] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.613s I0812 01:41:26.773977 14493 raft_consensus.cc:1677] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb1c732a162c403f9b391557aa6915c5 in term 1. I0812 01:41:26.775331 14544 leader_election.cc:347] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [CANDIDATE]: Term 1 election: Vote granted by peer 2b973a1abf534e0395b188635b112e01 I0812 01:41:26.775514 14374 raft_consensus.cc:1677] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb1c732a162c403f9b391557aa6915c5 in term 1. I0812 01:41:26.775593 14544 leader_election.cc:248] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0812 01:41:26.776221 14689 raft_consensus.cc:1970] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.621s I0812 01:41:26.776453 14544 leader_election.cc:347] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [CANDIDATE]: Term 1 election: Vote granted by peer 01494f0d9d974c5c86f7a027ae18de0e I0812 01:41:26.776458 14689 raft_consensus.cc:1859] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 1 FOLLOWER]: Leader election won for term 1 I0812 01:41:26.778231 14689 raft_consensus.cc:464] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 1 LEADER]: Becoming Leader. State: Replica: cb1c732a162c403f9b391557aa6915c5, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0812 01:41:26.778973 14689 consensus_queue.cc:145] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b973a1abf534e0395b188635b112e01" member_type: VOTER last_known_addr { host: "127.55.163.1" port: 54503 } } peers { permanent_uuid: "cb1c732a162c403f9b391557aa6915c5" member_type: VOTER last_known_addr { host: "127.55.163.2" port: 34073 } } peers { permanent_uuid: "01494f0d9d974c5c86f7a027ae18de0e" member_type: VOTER last_known_addr { host: "127.55.163.0" port: 49885 } } I0812 01:41:26.810109 14268 catalog_manager.cc:1982] T 5644aac95f434b23a28a109b37c3965a reported consensus state change: term changed from 0 to 1, leader changed from to cb1c732a162c403f9b391557aa6915c5 (127.55.163.2). New consensus state: current_term: 1 leader_uuid: "cb1c732a162c403f9b391557aa6915c5" config { opid_index: -1 peers { permanent_uuid: "2b973a1abf534e0395b188635b112e01" member_type: VOTER last_known_addr { host: "127.55.163.1" port: 54503 } } peers { permanent_uuid: "cb1c732a162c403f9b391557aa6915c5" member_type: VOTER last_known_addr { host: "127.55.163.2" port: 34073 } } peers { permanent_uuid: "01494f0d9d974c5c86f7a027ae18de0e" member_type: VOTER last_known_addr { host: "127.55.163.0" port: 49885 } } } W0812 01:41:26.811385 14694 raft_consensus_state.cc:541] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0812 01:41:26.906616 14374 raft_consensus.cc:869] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 1 FOLLOWER]: Refusing update from remote peer cb1c732a162c403f9b391557aa6915c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0812 01:41:26.906743 14493 raft_consensus.cc:869] T 5644aac95f434b23a28a109b37c3965a P 2b973a1abf534e0395b188635b112e01 [term 1 FOLLOWER]: Refusing update from remote peer cb1c732a162c403f9b391557aa6915c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0812 01:41:26.908359 14689 consensus_queue.cc:578] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [LEADER]: Connected to new peer: Peer: 01494f0d9d974c5c86f7a027ae18de0e, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0812 01:41:26.908706 14691 consensus_queue.cc:578] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [LEADER]: Connected to new peer: Peer: 2b973a1abf534e0395b188635b112e01, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0812 01:41:26.944756 14694 raft_consensus_state.cc:534] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 2 W0812 01:41:27.436498 14541 kernel_stack_watchdog.cc:130] Thread 14694 stuck at /data/10/dinesh/kudu/src/kudu/util/thread.cc:508 for 254ms: Kernel stack: [] futex_wait_queue_me+0xba/0xf0 [] futex_wait+0x1c0/0x310 [] do_futex+0x121/0xae0 [] sys_futex+0x7b/0x170 [] system_call_fastpath+0x16/0x1b [] 0xffffffffffffffff User stack: @ 0x425fb9 __tsan::ProcessPendingSignals() at ??:0 @ 0x44c53f pthread_create at /data/10/dinesh/kudu/thirdparty/llvm-3.8.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:289 @ 0x7fe613ae5811 kudu::Thread::StartThread() at ??:0 @ 0x7fe613af93f9 kudu::Thread::Create<>() at ??:0 @ 0x7fe613af5c9c kudu::ThreadPool::CreateThreadUnlocked() at ??:0 @ 0x7fe613af66cb kudu::ThreadPool::Submit() at ??:0 @ 0x7fe613af6323 kudu::ThreadPool::SubmitFunc() at ??:0 @ 0x7fe613af61e8 kudu::ThreadPool::SubmitClosure() at ??:0 @ 0x7fe618c896bd kudu::tablet::TransactionDriver::ApplyAsync() at ??:0 @ 0x7fe618c84f2b kudu::tablet::TransactionDriver::ReplicationFinished() at ??:0 @ 0x7fe618c7c61c kudu::internal::RunnableAdapter<>::Run() at ??:0 @ 0x7fe618c7c45c kudu::internal::InvokeHelper<>::MakeItSo() at ??:0 @ 0x7fe618c7c376 kudu::internal::Invoker<>::Run() at ??:0 @ 0x7fe619993d19 kudu::Callback<>::Run() at ??:0 @ 0x7fe6186f4e99 kudu::consensus::ConsensusRound::NotifyReplicationFinished() at ??:0 @ 0x7fe6187a3b97 kudu::consensus::ReplicaState::AdvanceCommittedIndexUnlocked() at ??:0 I0812 01:41:33.072803 14243 tablet_copy-itest.cc:326] Tombstoning follower tablet 5644aac95f434b23a28a109b37c3965a on TS 01494f0d9d974c5c86f7a027ae18de0e I0812 01:41:33.074645 14354 tablet_service.cc:658] Processing DeleteTablet for tablet 5644aac95f434b23a28a109b37c3965a with delete_type TABLET_DATA_TOMBSTONED from {real_user=dinesh, eff_user=} at 127.55.163.0:56330 I0812 01:41:33.075121 14354 tablet_peer.cc:214] Initiating TabletPeer shutdown for tablet: 5644aac95f434b23a28a109b37c3965a I0812 01:41:33.075845 14354 maintenance_manager.cc:176] Unregistered op CompactRowSetsOp(5644aac95f434b23a28a109b37c3965a) I0812 01:41:33.076033 14354 maintenance_manager.cc:176] Unregistered op MinorDeltaCompactionOp(5644aac95f434b23a28a109b37c3965a) I0812 01:41:33.076185 14354 maintenance_manager.cc:176] Unregistered op MajorDeltaCompactionOp(5644aac95f434b23a28a109b37c3965a) I0812 01:41:33.076416 14354 maintenance_manager.cc:176] Unregistered op FlushMRSOp(5644aac95f434b23a28a109b37c3965a) I0812 01:41:33.076551 14354 maintenance_manager.cc:176] Unregistered op FlushDeltaMemStoresOp(5644aac95f434b23a28a109b37c3965a) I0812 01:41:33.076668 14354 maintenance_manager.cc:176] Unregistered op LogGCOp(5644aac95f434b23a28a109b37c3965a) I0812 01:41:33.076828 14354 raft_consensus.cc:1495] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 1 FOLLOWER]: Raft consensus shutting down. I0812 01:41:33.077111 14354 raft_consensus.cc:1511] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e [term 1 FOLLOWER]: Raft consensus is shut down! I0812 01:41:33.086107 14354 ts_tablet_manager.cc:912] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0812 01:41:33.091351 14354 ts_tablet_manager.cc:922] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Tablet deleted. Last logged OpId: 1.31 I0812 01:41:33.092211 14354 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Deleted tablet blocks from disk I0812 01:41:33.454670 14691 consensus_peers.cc:303] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5 -> Peer 01494f0d9d974c5c86f7a027ae18de0e (127.55.163.0:49885): Sending request to start Tablet Copy I0812 01:41:33.456254 14374 ts_tablet_manager.cc:415] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Initiating tablet copy from Peer cb1c732a162c403f9b391557aa6915c5 (127.55.163.2:34073) I0812 01:41:33.458623 14374 tablet_copy_client.cc:161] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Tablet Copy client: Beginning tablet copy session from remote peer at address 127.55.163.2:34073 I0812 01:41:33.463841 14632 tablet_copy_service.cc:124] Beginning new tablet copy session on tablet 5644aac95f434b23a28a109b37c3965a from peer 01494f0d9d974c5c86f7a027ae18de0e at {real_user=dinesh, eff_user=} at 127.55.163.0:42446: session id = 01494f0d9d974c5c86f7a027ae18de0e-5644aac95f434b23a28a109b37c3965a I0812 01:41:33.467934 14632 tablet_copy_session.cc:142] T 5644aac95f434b23a28a109b37c3965a P cb1c732a162c403f9b391557aa6915c5: Tablet Copy: opened 0 blocks and 1 log segments I0812 01:41:33.473644 14374 ts_tablet_manager.cc:763] Registered tablet 5644aac95f434b23a28a109b37c3965a I0812 01:41:33.474030 14374 tablet_copy_client.cc:385] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Tablet Copy client: Starting download of 0 data blocks... I0812 01:41:33.474300 14374 tablet_copy_client.cc:350] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Tablet Copy client: Starting download of 1 WAL segments... I0812 01:41:33.474472 14374 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: TabletCopy: Downloading WAL segment with seq. number 1 (1/1) I0812 01:41:33.480816 14374 tablet_copy_client.cc:271] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Tablet Copy client: Tablet Copy complete. Replacing tablet superblock. I0812 01:41:33.480953 14374 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: TabletCopy: Replacing tablet superblock I0812 01:41:33.484431 14758 ts_tablet_manager.cc:629] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Bootstrapping tablet I0812 01:41:33.485124 14632 tablet_copy_service.cc:244] Request end of tablet copy session 01494f0d9d974c5c86f7a027ae18de0e-5644aac95f434b23a28a109b37c3965a received from {real_user=dinesh, eff_user=} at 127.55.163.0:42446 I0812 01:41:33.485288 14632 tablet_copy_service.cc:325] Ending tablet copy session 01494f0d9d974c5c86f7a027ae18de0e-5644aac95f434b23a28a109b37c3965a on tablet 5644aac95f434b23a28a109b37c3965a with peer 01494f0d9d974c5c86f7a027ae18de0e I0812 01:41:33.486490 14758 tablet_bootstrap.cc:400] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Bootstrap starting. I0812 01:41:33.488430 14758 tablet_bootstrap.cc:563] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Time spent opening tablet: real 0.001s user 0.000s sys 0.000s I0812 01:41:33.488981 14758 tablet_bootstrap.cc:619] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Will attempt to recover log segment /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/wals/5644aac95f434b23a28a109b37c3965a/wal-000000001 to /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/wals/5644aac95f434b23a28a109b37c3965a.recovery/wal-000000001 I0812 01:41:33.489095 14758 tablet_bootstrap.cc:627] T 5644aac95f434b23a28a109b37c3965a P 01494f0d9d974c5c86f7a027ae18de0e: Moving log directory /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/wals/5644aac95f434b23a28a109b37c3965a to recovery directory /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/wals/5644aac95f434b23a28a109b37c3965a.recovery in preparation for log replay W0812 01:41:33.489872 14758 log_util.cc:311] Could not read footer for segment: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/wals/5644aac95f434b23a28a109b37c3965a.recovery/wal-000000001: Not found: Footer not found. Footer magic doesn't match I0812 01:41:33.490037 14758 log_reader.cc:152] Log segment /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/wals/5644aac95f434b23a28a109b37c3965a.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data. I0812 01:41:33.497130 14243 tablet_copy-itest.cc:351] Test executed ListTablets RPCs during tablet copy. I0812 01:41:33.499727 14243 external_mini_cluster.cc:678] Killing /data/10/dinesh/kudu/build/tsan/bin/kudu-master with pid 14249 I0812 01:41:33.507179 14758 log_util.cc:359] Successfully rebuilt footer for segment: /tmp/kudutest-4519/tablet_copy-itest.TabletCopyITest.TestListTabletsDuringTabletCopy.1470991285217118-14243/minicluster-data/ts-0/wals/5644aac95f434b23a28a109b37c3965a.recovery/wal-000000001 (valid entries through byte offset 73034) I0812 01:41:33.507923 14758 tablet.cc:853] T 5644aac95f434b23a28a109b37c3965a Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0812 01:41:33.527546 14243 external_mini_cluster.cc:678] Killing /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver with pid 14293 I0812 01:41:33.576539 14243 external_mini_cluster.cc:678] Killing /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver with pid 14421 I0812 01:41:33.611004 14243 external_mini_cluster.cc:678] Killing /data/10/dinesh/kudu/build/tsan/bin/kudu-tserver with pid 14539 [ OK ] TabletCopyITest.TestListTabletsDuringTabletCopy (8444 ms) [----------] 1 test from TabletCopyITest (8444 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (8444 ms total) [ PASSED ] 1 test.