[ RUN ] MasterFailoverTest.TestRenameTableSync I0225 21:50:39.039475 26122 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-0 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-0 --rpc_bind_addresses=127.0.0.1:11010 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-0 --leader_failure_max_missed_heartbeat_periods=1.0 --master_addresses=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --enable_leader_failure_detection=true --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 21:50:39.343858 1149 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 21:50:39.344655 1149 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 21:50:39.354418 1149 master_main.cc:58] Initializing master server... I0225 21:50:39.355356 1149 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 283948 I0225 21:50:39.361176 1149 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-0/instance: No such file or directory (error 2) I0225 21:50:39.361260 1149 server_base.cc:165] Creating new FS layout I0225 21:50:39.377742 1149 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-0/instance: uuid: "7d1972008325413a943c1ee4f140ee66" format_stamp: "Formatted at 2016-02-26 05:50:39 on ve0518.halxg.cloudera.com" I0225 21:50:39.390677 1149 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-0 uuid: "7d1972008325413a943c1ee4f140ee66" format_stamp: "Formatted at 2016-02-26 05:50:39 on ve0518.halxg.cloudera.com" I0225 21:50:39.415994 1149 master_main.cc:61] Starting Master server... I0225 21:50:39.453559 1149 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:11010 I0225 21:50:39.454601 1149 webserver.cc:135] Starting webserver on localhost:0 I0225 21:50:39.454655 1149 webserver.cc:146] Document root disabled I0225 21:50:39.455515 1149 net_util.cc:175] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0225 21:50:39.457451 1149 webserver.cc:229] Webserver started. Bound to: http://127.0.0.1:59454/ I0225 21:50:39.458875 1149 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-0/info.pb I0225 21:50:39.466436 26122 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-master as pid 1149 I0225 21:50:39.467022 26122 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-1 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-1 --rpc_bind_addresses=127.0.0.1:11011 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-1 --leader_failure_max_missed_heartbeat_periods=1.0 --master_addresses=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --enable_leader_failure_detection=true --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 21:50:39.479771 1190 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:39.483392 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11010, user_credentials={real_user=adar, eff_user=}} I0225 21:50:39.506592 1190 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:39.507064 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 21:50:39.509080 1190 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 21:50:39.569576 1190 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 1 W0225 21:50:39.569964 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 21:50:39.571132 1190 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 21:50:39.622349 1190 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 2 W0225 21:50:39.622730 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 21:50:39.623719 1190 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 21:50:39.725991 1190 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 3 W0225 21:50:39.726284 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 21:50:39.727452 1190 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 21:50:39.782444 1191 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 21:50:39.783257 1191 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 21:50:39.794684 1191 master_main.cc:58] Initializing master server... I0225 21:50:39.795666 1191 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 283948 I0225 21:50:39.801725 1191 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-1/instance: No such file or directory (error 2) I0225 21:50:39.801834 1191 server_base.cc:165] Creating new FS layout I0225 21:50:39.810678 1191 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-1/instance: uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" format_stamp: "Formatted at 2016-02-26 05:50:39 on ve0518.halxg.cloudera.com" I0225 21:50:39.823899 1191 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-1 uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" format_stamp: "Formatted at 2016-02-26 05:50:39 on ve0518.halxg.cloudera.com" I0225 21:50:39.851778 1191 master_main.cc:61] Starting Master server... I0225 21:50:39.856672 1190 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 4 W0225 21:50:39.857030 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 21:50:39.858119 1190 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 21:50:39.890036 1191 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:11011 I0225 21:50:39.891088 1191 webserver.cc:135] Starting webserver on localhost:0 I0225 21:50:39.891132 1191 webserver.cc:146] Document root disabled I0225 21:50:39.892251 1191 net_util.cc:175] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0225 21:50:39.893800 1191 webserver.cc:229] Webserver started. Bound to: http://127.0.0.1:50573/ I0225 21:50:39.895916 1191 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-1/info.pb I0225 21:50:39.897377 26122 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-master as pid 1191 I0225 21:50:39.897965 26122 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-2 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-2 --rpc_bind_addresses=127.0.0.1:11012 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-2 --leader_failure_max_missed_heartbeat_periods=1.0 --master_addresses=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --enable_leader_failure_detection=true --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 21:50:39.917078 1235 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:39.920383 1199 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11010, user_credentials={real_user=adar, eff_user=}} I0225 21:50:39.936769 1235 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:39.937206 1199 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} I0225 21:50:39.947391 1235 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:39.947840 1199 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 21:50:39.949671 1235 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 21:50:40.010136 1235 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 1 W0225 21:50:40.010546 1199 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 21:50:40.011765 1235 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 21:50:40.063012 1235 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 2 W0225 21:50:40.063354 1199 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 21:50:40.064599 1235 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 21:50:40.140446 1190 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 5 W0225 21:50:40.140975 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} I0225 21:50:40.145071 1190 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:40.145567 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 21:50:40.146522 1190 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 21:50:40.166864 1235 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 3 W0225 21:50:40.167189 1199 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 21:50:40.168445 1235 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 21:50:40.195816 1190 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 1 W0225 21:50:40.196358 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 21:50:40.197634 1190 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 21:50:40.234908 1236 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 21:50:40.235812 1236 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 21:50:40.245848 1236 master_main.cc:58] Initializing master server... I0225 21:50:40.246851 1236 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 284448 I0225 21:50:40.248963 1190 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 2 W0225 21:50:40.249462 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 21:50:40.250795 1190 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 21:50:40.253115 1236 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-2/instance: No such file or directory (error 2) I0225 21:50:40.253203 1236 server_base.cc:165] Creating new FS layout I0225 21:50:40.261927 1236 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-2/instance: uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" format_stamp: "Formatted at 2016-02-26 05:50:40 on ve0518.halxg.cloudera.com" I0225 21:50:40.275166 1236 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-2 uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" format_stamp: "Formatted at 2016-02-26 05:50:40 on ve0518.halxg.cloudera.com" I0225 21:50:40.297657 1235 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 4 W0225 21:50:40.297991 1199 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 21:50:40.299134 1235 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 21:50:40.302072 1236 master_main.cc:61] Starting Master server... I0225 21:50:40.316622 1236 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:11012 I0225 21:50:40.317458 1236 webserver.cc:135] Starting webserver on localhost:0 I0225 21:50:40.317507 1236 webserver.cc:146] Document root disabled I0225 21:50:40.318229 1236 net_util.cc:175] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0225 21:50:40.319051 1236 webserver.cc:229] Webserver started. Bound to: http://127.0.0.1:41241/ I0225 21:50:40.320194 1236 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/master-2/info.pb I0225 21:50:40.328477 26122 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-master as pid 1236 I0225 21:50:40.329296 26122 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-0 --rpc_bind_addresses=127.102.10.0:0 --local_ip_for_outbound_sockets=127.102.10.0 --webserver_interface=127.102.10.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-0 --heartbeat_rpc_timeout_ms=1000 --heartbeat_max_failures_before_backoff=1 --heartbeat_interval_ms=500 --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 21:50:40.336346 1280 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:40.339689 1244 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11010, user_credentials={real_user=adar, eff_user=}} I0225 21:50:40.348063 1190 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 3 W0225 21:50:40.348438 1154 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} I0225 21:50:40.354152 1280 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:40.355485 1244 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} I0225 21:50:40.358572 1280 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } has no permanent_uuid. Determining permanent_uuid... W0225 21:50:40.358976 1244 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} I0225 21:50:40.363456 1190 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66: Bootstrap starting. I0225 21:50:40.366338 1190 tablet_bootstrap.cc:524] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0225 21:50:40.366646 1190 tablet_bootstrap.cc:464] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66: No blocks or log segments found. Creating new log. I0225 21:50:40.367517 1190 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 21:50:40.368610 1280 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4: Bootstrap starting. I0225 21:50:40.370098 1190 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66: No bootstrap required, opened a new log I0225 21:50:40.373117 1280 tablet_bootstrap.cc:524] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0225 21:50:40.373391 1190 raft_consensus.cc:268] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:40.373677 1280 tablet_bootstrap.cc:464] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4: No blocks or log segments found. Creating new log. I0225 21:50:40.374250 1190 raft_consensus.cc:300] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 21:50:40.374409 1190 raft_consensus.cc:477] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1972008325413a943c1ee4f140ee66, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:40.374584 1190 consensus_queue.cc:162] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 21:50:40.375191 1190 sys_catalog.cc:227] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:40.375322 1190 sys_catalog.cc:230] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:40.375545 1280 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 21:50:40.375412 1288 sys_catalog.cc:227] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:40.375768 1288 sys_catalog.cc:230] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:40.376571 1190 sys_catalog.cc:299] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: configured and running, proceeding with master startup. I0225 21:50:40.377085 1149 master_main.cc:64] Master server successfully started. I0225 21:50:40.379340 1280 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4: No bootstrap required, opened a new log I0225 21:50:40.385623 1280 raft_consensus.cc:268] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:40.387241 1280 raft_consensus.cc:300] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 21:50:40.387523 1280 raft_consensus.cc:477] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c72ab184ce64d999dd3ebf9e3720cd4, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:40.387897 1280 consensus_queue.cc:162] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 21:50:40.388876 1280 sys_catalog.cc:227] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:40.389255 1280 sys_catalog.cc:230] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:40.389000 1293 sys_catalog.cc:227] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:40.389463 1293 sys_catalog.cc:230] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:40.391613 1280 sys_catalog.cc:299] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [sys.catalog]: configured and running, proceeding with master startup. I0225 21:50:40.392431 1236 master_main.cc:64] Master server successfully started. I0225 21:50:40.581331 1235 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 5 W0225 21:50:40.581712 1199 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} I0225 21:50:40.589882 1235 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70: Bootstrap starting. I0225 21:50:40.593801 1235 tablet_bootstrap.cc:524] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70: Time spent opening tablet: real 0.001s user 0.002s sys 0.000s I0225 21:50:40.594307 1235 tablet_bootstrap.cc:464] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70: No blocks or log segments found. Creating new log. I0225 21:50:40.595957 1235 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 21:50:40.599844 1235 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70: No bootstrap required, opened a new log I0225 21:50:40.605901 1235 raft_consensus.cc:268] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:40.607437 1235 raft_consensus.cc:300] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 21:50:40.607715 1235 raft_consensus.cc:477] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbd1640a4934d2e9bbf6d3a52aeba70, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:40.608064 1235 consensus_queue.cc:162] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 21:50:40.609097 1235 sys_catalog.cc:227] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:40.609411 1235 sys_catalog.cc:230] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:40.609261 1298 sys_catalog.cc:227] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:40.609658 1298 sys_catalog.cc:230] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:40.611614 1235 sys_catalog.cc:299] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: configured and running, proceeding with master startup. I0225 21:50:40.612509 1191 master_main.cc:64] Master server successfully started. I0225 21:50:40.615839 1282 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 21:50:40.616168 1282 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 21:50:40.623818 1282 tablet_server_main.cc:54] Initializing tablet server... I0225 21:50:40.626015 1282 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 284448 I0225 21:50:40.629705 1282 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-0/instance: No such file or directory (error 2) I0225 21:50:40.629808 1282 server_base.cc:165] Creating new FS layout I0225 21:50:40.636783 1282 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-0/instance: uuid: "5a9fdcf0227045cf9f868590152defd9" format_stamp: "Formatted at 2016-02-26 05:50:40 on ve0518.halxg.cloudera.com" I0225 21:50:40.645699 1282 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-0 uuid: "5a9fdcf0227045cf9f868590152defd9" format_stamp: "Formatted at 2016-02-26 05:50:40 on ve0518.halxg.cloudera.com" I0225 21:50:40.658444 1282 tablet_server_main.cc:57] Starting tablet server... I0225 21:50:40.691444 1282 rpc_server.cc:164] RPC server started. Bound to: 127.102.10.0:41575 I0225 21:50:40.692068 1282 webserver.cc:135] Starting webserver on 127.102.10.0:0 I0225 21:50:40.692101 1282 webserver.cc:146] Document root disabled I0225 21:50:40.692939 1282 webserver.cc:229] Webserver started. Bound to: http://127.102.10.0:53752/ I0225 21:50:40.693727 1282 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-0/info.pb I0225 21:50:40.694653 1282 tablet_server_main.cc:60] Tablet server successfully started. I0225 21:50:40.700120 26122 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-tserver as pid 1282 I0225 21:50:40.700906 26122 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-1 --rpc_bind_addresses=127.102.10.1:0 --local_ip_for_outbound_sockets=127.102.10.1 --webserver_interface=127.102.10.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-1 --heartbeat_rpc_timeout_ms=1000 --heartbeat_max_failures_before_backoff=1 --heartbeat_interval_ms=500 --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 21:50:40.740005 1287 raft_consensus.cc:362] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0225 21:50:40.740176 1287 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 0 FOLLOWER]: Advancing to term 1 I0225 21:50:40.742012 1287 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.506s I0225 21:50:40.742163 1287 raft_consensus.cc:378] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:40.745829 1287 leader_election.cc:222] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 1 election: Requesting vote from peer 5cbd1640a4934d2e9bbf6d3a52aeba70 I0225 21:50:40.746448 1287 leader_election.cc:222] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 1 election: Requesting vote from peer 4c72ab184ce64d999dd3ebf9e3720cd4 I0225 21:50:40.748194 1261 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 0 FOLLOWER]: Advancing to term 1 I0225 21:50:40.748213 1216 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 0 FOLLOWER]: Advancing to term 1 I0225 21:50:40.750000 1261 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.519s I0225 21:50:40.750237 1216 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.529s I0225 21:50:40.751902 1216 raft_consensus.cc:1649] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1972008325413a943c1ee4f140ee66 in term 1. I0225 21:50:40.752008 1261 raft_consensus.cc:1649] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1972008325413a943c1ee4f140ee66 in term 1. I0225 21:50:40.752303 1292 raft_consensus.cc:362] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 1 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0225 21:50:40.752475 1292 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 1 FOLLOWER]: Advancing to term 2 I0225 21:50:40.752817 1154 leader_election.cc:346] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 1 election: Vote granted by peer 5cbd1640a4934d2e9bbf6d3a52aeba70 I0225 21:50:40.752995 1154 leader_election.cc:247] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0225 21:50:40.753490 1154 leader_election.cc:346] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 1 election: Vote granted by peer 4c72ab184ce64d999dd3ebf9e3720cd4 I0225 21:50:40.753489 1288 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.515s I0225 21:50:40.753712 1288 raft_consensus.cc:1831] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 FOLLOWER]: Leader election won for term 1 I0225 21:50:40.754395 1292 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.567s I0225 21:50:40.754470 1292 raft_consensus.cc:378] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:40.754657 1288 raft_consensus.cc:442] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 LEADER]: Becoming Leader. State: Replica: 7d1972008325413a943c1ee4f140ee66, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:40.755425 1288 consensus_queue.cc:145] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:40.756641 1292 leader_election.cc:222] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [CANDIDATE]: Term 2 election: Requesting vote from peer 7d1972008325413a943c1ee4f140ee66 I0225 21:50:40.756989 1292 leader_election.cc:222] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [CANDIDATE]: Term 2 election: Requesting vote from peer 5cbd1640a4934d2e9bbf6d3a52aeba70 I0225 21:50:40.758179 1217 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 1 FOLLOWER]: Advancing to term 2 I0225 21:50:40.759774 1217 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.545s I0225 21:50:40.760959 1217 raft_consensus.cc:1649] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c72ab184ce64d999dd3ebf9e3720cd4 in term 2. I0225 21:50:40.761456 1244 leader_election.cc:346] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [CANDIDATE]: Term 2 election: Vote granted by peer 5cbd1640a4934d2e9bbf6d3a52aeba70 I0225 21:50:40.761617 1244 leader_election.cc:247] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0225 21:50:40.762054 1293 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.598s I0225 21:50:40.762289 1293 raft_consensus.cc:1831] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 2 FOLLOWER]: Leader election won for term 2 I0225 21:50:40.763198 1293 raft_consensus.cc:442] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 2 LEADER]: Becoming Leader. State: Replica: 4c72ab184ce64d999dd3ebf9e3720cd4, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:40.763921 1293 consensus_queue.cc:145] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:40.767066 1171 raft_consensus.cc:1612] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 4c72ab184ce64d999dd3ebf9e3720cd4 for term 2 because replica is either leader or believes a valid leader to be alive. I0225 21:50:40.766782 1418 sys_catalog.cc:227] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1972008325413a943c1ee4f140ee66. Latest consensus state: current_term: 1 leader_uuid: "7d1972008325413a943c1ee4f140ee66" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:40.767283 1418 sys_catalog.cc:230] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER I0225 21:50:40.768642 1424 catalog_manager.cc:520] Loading table and tablet metadata into memory... I0225 21:50:40.768553 1244 leader_election.cc:360] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [CANDIDATE]: Term 2 election: Vote denied by peer 7d1972008325413a943c1ee4f140ee66. Message: Invalid argument: T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 4c72ab184ce64d999dd3ebf9e3720cd4 for term 2 because replica is either leader or believes a valid leader to be alive. W0225 21:50:40.769250 1425 raft_consensus_state.cc:524] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0225 21:50:40.770012 1411 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 21:50:40.770390 1411 heartbeater.cc:328] Registering TS with master... I0225 21:50:40.770689 1411 heartbeater.cc:334] Sending a full tablet report to master... W0225 21:50:40.772280 1166 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Service unavailable: Leader not yet ready to serve requests W0225 21:50:40.773232 1411 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11010: Service unavailable: master is no longer the leader W0225 21:50:40.773357 1411 heartbeater.cc:287] Failed 1 heartbeats in a row: no longer allowing fast heartbeat attempts. I0225 21:50:40.776064 1421 sys_catalog.cc:227] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c72ab184ce64d999dd3ebf9e3720cd4. Latest consensus state: current_term: 2 leader_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:40.776386 1421 sys_catalog.cc:230] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER I0225 21:50:40.776952 1411 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 21:50:40.777045 1411 heartbeater.cc:328] Registering TS with master... I0225 21:50:40.777144 1411 heartbeater.cc:334] Sending a full tablet report to master... I0225 21:50:40.777802 1427 catalog_manager.cc:520] Loading table and tablet metadata into memory... W0225 21:50:40.777907 1428 raft_consensus_state.cc:524] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 2 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 2 I0225 21:50:40.778889 1159 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" instance_seqno: 1456465840685239 } with Master I0225 21:50:40.965801 1417 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 21:50:40.966141 1417 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 21:50:40.974256 1417 tablet_server_main.cc:54] Initializing tablet server... I0225 21:50:40.976405 1417 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 284448 I0225 21:50:40.980372 1417 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-1/instance: No such file or directory (error 2) I0225 21:50:40.980455 1417 server_base.cc:165] Creating new FS layout I0225 21:50:40.987660 1417 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-1/instance: uuid: "7628b292019d4388b096a4f96b2283eb" format_stamp: "Formatted at 2016-02-26 05:50:40 on ve0518.halxg.cloudera.com" I0225 21:50:40.997555 1417 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-1 uuid: "7628b292019d4388b096a4f96b2283eb" format_stamp: "Formatted at 2016-02-26 05:50:40 on ve0518.halxg.cloudera.com" I0225 21:50:41.010951 1417 tablet_server_main.cc:57] Starting tablet server... I0225 21:50:41.040797 1417 rpc_server.cc:164] RPC server started. Bound to: 127.102.10.1:36483 I0225 21:50:41.041478 1417 webserver.cc:135] Starting webserver on 127.102.10.1:0 I0225 21:50:41.041522 1417 webserver.cc:146] Document root disabled I0225 21:50:41.042407 1417 webserver.cc:229] Webserver started. Bound to: http://127.102.10.1:36445/ I0225 21:50:41.043351 1417 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-1/info.pb I0225 21:50:41.044315 1417 tablet_server_main.cc:60] Tablet server successfully started. I0225 21:50:41.049928 26122 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-tserver as pid 1417 I0225 21:50:41.050712 26122 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-2 --rpc_bind_addresses=127.102.10.2:0 --local_ip_for_outbound_sockets=127.102.10.2 --webserver_interface=127.102.10.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-2 --heartbeat_rpc_timeout_ms=1000 --heartbeat_max_failures_before_backoff=1 --heartbeat_interval_ms=500 --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 21:50:41.059325 1541 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 21:50:41.059535 1541 heartbeater.cc:328] Registering TS with master... I0225 21:50:41.059696 1541 heartbeater.cc:334] Sending a full tablet report to master... I0225 21:50:41.060894 1162 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" instance_seqno: 1456465841035103 } with Master I0225 21:50:41.261396 1262 raft_consensus.cc:799] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 2 LEADER]: Rejecting Update request from peer 7d1972008325413a943c1ee4f140ee66 for earlier term 1. Current term is 2. Ops: [] I0225 21:50:41.261461 1218 raft_consensus.cc:799] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 2 FOLLOWER]: Rejecting Update request from peer 7d1972008325413a943c1ee4f140ee66 for earlier term 1. Current term is 2. Ops: [] I0225 21:50:41.263535 1288 consensus_queue.cc:582] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [LEADER]: Peer responded invalid term: Peer: 4c72ab184ce64d999dd3ebf9e3720cd4, Is new: false, Last received: 2.1, Next index: 2, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 21:50:41.264027 1418 consensus_queue.cc:582] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [LEADER]: Peer responded invalid term: Peer: 5cbd1640a4934d2e9bbf6d3a52aeba70, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 21:50:41.264369 1425 raft_consensus.cc:1983] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 LEADER]: Stepping down as leader of term 1 I0225 21:50:41.264482 1425 raft_consensus.cc:477] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d1972008325413a943c1ee4f140ee66, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 1 Committed: term: 0 index: 0} I0225 21:50:41.264950 1425 consensus_queue.cc:162] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 1.1, Current term: 1, Majority size: -1, State: 1, Mode: NON_LEADER I0225 21:50:41.265669 1425 consensus_peers.cc:342] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Closing peer: 4c72ab184ce64d999dd3ebf9e3720cd4 I0225 21:50:41.266325 1425 consensus_peers.cc:342] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 5cbd1640a4934d2e9bbf6d3a52aeba70 (127.0.0.1:11011): Closing peer: 5cbd1640a4934d2e9bbf6d3a52aeba70 I0225 21:50:41.266906 1425 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 1 FOLLOWER]: Advancing to term 2 I0225 21:50:41.268551 1219 raft_consensus.cc:831] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 2 FOLLOWER]: Refusing update from remote peer 4c72ab184ce64d999dd3ebf9e3720cd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I0225 21:50:41.269493 1172 raft_consensus.cc:831] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 2 FOLLOWER]: Refusing update from remote peer 4c72ab184ce64d999dd3ebf9e3720cd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch) I0225 21:50:41.269649 1172 raft_consensus_state.cc:397] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 2 FOLLOWER]: Aborting all transactions after (but not including): 0. Current State: Replica: 7d1972008325413a943c1ee4f140ee66, State: 1, Role: FOLLOWER Watermarks: {Received: term: 1 index: 1 Committed: term: 0 index: 0} I0225 21:50:41.269670 1293 consensus_queue.cc:572] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [LEADER]: Connected to new peer: Peer: 5cbd1640a4934d2e9bbf6d3a52aeba70, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 21:50:41.269825 1172 raft_consensus_state.cc:423] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 2 FOLLOWER]: Aborting uncommitted operation due to leader change: 1.1 I0225 21:50:41.270074 1172 raft_consensus.cc:1877] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66: NO_OP replication failed: Aborted: Transaction aborted by new leader I0225 21:50:41.271257 1421 consensus_queue.cc:572] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [LEADER]: Connected to new peer: Peer: 7d1972008325413a943c1ee4f140ee66, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 21:50:41.275373 1288 sys_catalog.cc:227] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c72ab184ce64d999dd3ebf9e3720cd4. Latest consensus state: current_term: 2 leader_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:41.275853 1288 sys_catalog.cc:230] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:41.278293 1428 raft_consensus_state.cc:517] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 [term 2 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 2 index: 1 I0225 21:50:41.280992 1418 sys_catalog.cc:227] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:41.281375 1418 sys_catalog.cc:230] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER W0225 21:50:41.282291 1163 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Illegal state: Not the leader. Local UUID: 7d1972008325413a943c1ee4f140ee66, Consensus state: current_term: 2 leader_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } W0225 21:50:41.283093 1411 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11010: Service unavailable: master is no longer the leader I0225 21:50:41.284059 1548 sys_catalog.cc:227] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c72ab184ce64d999dd3ebf9e3720cd4. Latest consensus state: current_term: 2 leader_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:41.284414 1548 sys_catalog.cc:230] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:41.289101 1548 sys_catalog.cc:227] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:41.289398 1548 sys_catalog.cc:230] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:41.382838 1547 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 21:50:41.383180 1547 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 21:50:41.393101 1547 tablet_server_main.cc:54] Initializing tablet server... I0225 21:50:41.395814 1547 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 284948 I0225 21:50:41.400815 1547 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-2/instance: No such file or directory (error 2) I0225 21:50:41.400931 1547 server_base.cc:165] Creating new FS layout I0225 21:50:41.409461 1547 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-2/instance: uuid: "abbe08b58a6b4e30b0250bd1288a9f32" format_stamp: "Formatted at 2016-02-26 05:50:41 on ve0518.halxg.cloudera.com" I0225 21:50:41.424156 1547 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-2 uuid: "abbe08b58a6b4e30b0250bd1288a9f32" format_stamp: "Formatted at 2016-02-26 05:50:41 on ve0518.halxg.cloudera.com" I0225 21:50:41.444057 1547 tablet_server_main.cc:57] Starting tablet server... I0225 21:50:41.480437 1547 rpc_server.cc:164] RPC server started. Bound to: 127.102.10.2:49416 I0225 21:50:41.481381 1547 webserver.cc:135] Starting webserver on 127.102.10.2:0 I0225 21:50:41.481431 1547 webserver.cc:146] Document root disabled I0225 21:50:41.482575 1547 webserver.cc:229] Webserver started. Bound to: http://127.102.10.2:36626/ I0225 21:50:41.483790 1547 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456460284463433-26122/minicluster-data/ts-2/info.pb I0225 21:50:41.485038 1547 tablet_server_main.cc:60] Tablet server successfully started. I0225 21:50:41.490203 26122 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-tserver as pid 1547 I0225 21:50:41.501610 1663 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11012 I0225 21:50:41.501855 1663 heartbeater.cc:328] Registering TS with master... I0225 21:50:41.502053 1663 heartbeater.cc:334] Sending a full tablet report to master... I0225 21:50:41.504115 1249 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" instance_seqno: 1456465841472939 } with Master W0225 21:50:41.563549 1165 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Illegal state: Not the leader. Local UUID: 7d1972008325413a943c1ee4f140ee66, Consensus state: current_term: 2 leader_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } W0225 21:50:41.564479 1541 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11010: Service unavailable: master is no longer the leader I0225 21:50:41.788573 1411 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11012 I0225 21:50:41.790033 1252 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" instance_seqno: 1456465840685239 } as {real_user=adar, eff_user=} at 127.102.10.0:44682; Asking this server to re-register. I0225 21:50:41.790983 1411 heartbeater.cc:328] Registering TS with master... I0225 21:50:41.791165 1411 heartbeater.cc:334] Sending a full tablet report to master... I0225 21:50:41.792407 1254 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" instance_seqno: 1456465840685239 } with Master I0225 21:50:42.069000 1541 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11012 I0225 21:50:42.070344 1257 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" instance_seqno: 1456465841035103 } as {real_user=adar, eff_user=} at 127.102.10.1:47953; Asking this server to re-register. I0225 21:50:42.071209 1541 heartbeater.cc:328] Registering TS with master... I0225 21:50:42.071378 1541 heartbeater.cc:334] Sending a full tablet report to master... I0225 21:50:42.072691 1248 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" instance_seqno: 1456465841035103 } with Master I0225 21:50:42.076535 26122 external_mini_cluster.cc:318] 3 TS(s) registered with Master I0225 21:50:42.094878 1252 catalog_manager.cc:678] CreateTable from {real_user=adar, eff_user=} at 127.0.0.1:57125: name: "testAlterTableSync" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } split_rows { } partition_schema { } I0225 21:50:42.157717 1459 tablet_service.cc:602] Processing CreateTablet for tablet a4730e4d785a49069ba2bb382ede3f9e (table=testAlterTableSync [id=93c0db77fc8447458439412b937c94d3]), partition=range: [(), ()) I0225 21:50:42.157979 1329 tablet_service.cc:602] Processing CreateTablet for tablet a4730e4d785a49069ba2bb382ede3f9e (table=testAlterTableSync [id=93c0db77fc8447458439412b937c94d3]), partition=range: [(), ()) I0225 21:50:42.158257 1580 tablet_service.cc:602] Processing CreateTablet for tablet a4730e4d785a49069ba2bb382ede3f9e (table=testAlterTableSync [id=93c0db77fc8447458439412b937c94d3]), partition=range: [(), ()) I0225 21:50:42.170214 1459 ts_tablet_manager.cc:744] Registered tablet a4730e4d785a49069ba2bb382ede3f9e I0225 21:50:42.170817 1691 ts_tablet_manager.cc:616] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb: Bootstrapping tablet I0225 21:50:42.172603 1691 tablet_bootstrap.cc:363] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb: Bootstrap starting. I0225 21:50:42.173172 1329 ts_tablet_manager.cc:744] Registered tablet a4730e4d785a49069ba2bb382ede3f9e I0225 21:50:42.174093 1693 ts_tablet_manager.cc:616] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9: Bootstrapping tablet I0225 21:50:42.174309 1580 ts_tablet_manager.cc:744] Registered tablet a4730e4d785a49069ba2bb382ede3f9e I0225 21:50:42.175199 1694 ts_tablet_manager.cc:616] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32: Bootstrapping tablet I0225 21:50:42.176941 1691 tablet_bootstrap.cc:524] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb: Time spent opening tablet: real 0.001s user 0.002s sys 0.000s I0225 21:50:42.176939 1693 tablet_bootstrap.cc:363] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9: Bootstrap starting. I0225 21:50:42.177381 1691 tablet_bootstrap.cc:464] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb: No blocks or log segments found. Creating new log. I0225 21:50:42.177924 1694 tablet_bootstrap.cc:363] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32: Bootstrap starting. I0225 21:50:42.178932 1691 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 21:50:42.181608 1693 tablet_bootstrap.cc:524] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0225 21:50:42.182186 1693 tablet_bootstrap.cc:464] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9: No blocks or log segments found. Creating new log. I0225 21:50:42.182533 1691 tablet_bootstrap.cc:363] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb: No bootstrap required, opened a new log I0225 21:50:42.182867 1694 tablet_bootstrap.cc:524] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32: Time spent opening tablet: real 0.002s user 0.002s sys 0.000s I0225 21:50:42.182971 1691 ts_tablet_manager.cc:621] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.002s I0225 21:50:42.183364 1694 tablet_bootstrap.cc:464] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32: No blocks or log segments found. Creating new log. I0225 21:50:42.184109 1693 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 21:50:42.184607 1694 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 21:50:42.187731 1694 tablet_bootstrap.cc:363] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32: No bootstrap required, opened a new log I0225 21:50:42.187958 1693 tablet_bootstrap.cc:363] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9: No bootstrap required, opened a new log I0225 21:50:42.188077 1694 ts_tablet_manager.cc:621] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s I0225 21:50:42.188393 1693 ts_tablet_manager.cc:621] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s I0225 21:50:42.188343 1691 raft_consensus.cc:268] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" member_type: VOTER last_known_addr { host: "127.102.10.1" port: 36483 } } peers { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" member_type: VOTER last_known_addr { host: "127.102.10.2" port: 49416 } } peers { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" member_type: VOTER last_known_addr { host: "127.102.10.0" port: 41575 } } I0225 21:50:42.190151 1691 raft_consensus.cc:300] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 21:50:42.190412 1691 raft_consensus.cc:477] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7628b292019d4388b096a4f96b2283eb, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:42.190721 1691 consensus_queue.cc:162] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 21:50:42.192845 1691 ts_tablet_manager.cc:643] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s I0225 21:50:42.192476 1694 raft_consensus.cc:268] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" member_type: VOTER last_known_addr { host: "127.102.10.1" port: 36483 } } peers { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" member_type: VOTER last_known_addr { host: "127.102.10.2" port: 49416 } } peers { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" member_type: VOTER last_known_addr { host: "127.102.10.0" port: 41575 } } I0225 21:50:42.193980 1694 raft_consensus.cc:300] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 21:50:42.194193 1694 raft_consensus.cc:477] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbe08b58a6b4e30b0250bd1288a9f32, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:42.194456 1694 consensus_queue.cc:162] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 21:50:42.194581 1693 raft_consensus.cc:268] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" member_type: VOTER last_known_addr { host: "127.102.10.1" port: 36483 } } peers { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" member_type: VOTER last_known_addr { host: "127.102.10.2" port: 49416 } } peers { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" member_type: VOTER last_known_addr { host: "127.102.10.0" port: 41575 } } I0225 21:50:42.196193 1694 ts_tablet_manager.cc:643] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s I0225 21:50:42.196689 1693 raft_consensus.cc:300] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 21:50:42.196993 1693 raft_consensus.cc:477] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9fdcf0227045cf9f868590152defd9, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:42.197347 1693 consensus_queue.cc:162] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 21:50:42.200031 1693 ts_tablet_manager.cc:643] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s I0225 21:50:42.464124 1704 raft_consensus.cc:362] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0225 21:50:42.464357 1704 raft_consensus.cc:1988] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 0 FOLLOWER]: Advancing to term 1 I0225 21:50:42.466421 1704 raft_consensus.cc:1939] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.540s I0225 21:50:42.466603 1704 raft_consensus.cc:378] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" member_type: VOTER last_known_addr { host: "127.102.10.1" port: 36483 } } peers { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" member_type: VOTER last_known_addr { host: "127.102.10.2" port: 49416 } } peers { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" member_type: VOTER last_known_addr { host: "127.102.10.0" port: 41575 } } I0225 21:50:42.470563 1704 leader_election.cc:222] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [CANDIDATE]: Term 1 election: Requesting vote from peer 7628b292019d4388b096a4f96b2283eb I0225 21:50:42.471213 1704 leader_election.cc:222] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [CANDIDATE]: Term 1 election: Requesting vote from peer abbe08b58a6b4e30b0250bd1288a9f32 I0225 21:50:42.475834 1479 raft_consensus.cc:1988] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb [term 0 FOLLOWER]: Advancing to term 1 I0225 21:50:42.476604 1600 raft_consensus.cc:1988] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32 [term 0 FOLLOWER]: Advancing to term 1 I0225 21:50:42.477881 1479 raft_consensus.cc:1939] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.592s I0225 21:50:42.478664 1600 raft_consensus.cc:1939] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.592s I0225 21:50:42.479521 1479 raft_consensus.cc:1649] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a9fdcf0227045cf9f868590152defd9 in term 1. I0225 21:50:42.480214 1600 raft_consensus.cc:1649] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a9fdcf0227045cf9f868590152defd9 in term 1. I0225 21:50:42.480916 1303 leader_election.cc:346] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [CANDIDATE]: Term 1 election: Vote granted by peer 7628b292019d4388b096a4f96b2283eb I0225 21:50:42.481209 1302 leader_election.cc:346] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [CANDIDATE]: Term 1 election: Vote granted by peer abbe08b58a6b4e30b0250bd1288a9f32 I0225 21:50:42.481319 1302 leader_election.cc:247] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0225 21:50:42.481784 1706 raft_consensus.cc:1939] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.569s I0225 21:50:42.482058 1706 raft_consensus.cc:1831] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 1 FOLLOWER]: Leader election won for term 1 I0225 21:50:42.483196 1706 raft_consensus.cc:442] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 1 LEADER]: Becoming Leader. State: Replica: 5a9fdcf0227045cf9f868590152defd9, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 21:50:42.484052 1706 consensus_queue.cc:145] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" member_type: VOTER last_known_addr { host: "127.102.10.1" port: 36483 } } peers { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" member_type: VOTER last_known_addr { host: "127.102.10.2" port: 49416 } } peers { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" member_type: VOTER last_known_addr { host: "127.102.10.0" port: 41575 } } I0225 21:50:42.495075 1249 catalog_manager.cc:1558] Tablet: a4730e4d785a49069ba2bb382ede3f9e reported consensus state change. New consensus state: current_term: 1 leader_uuid: "5a9fdcf0227045cf9f868590152defd9" config { opid_index: -1 local: false peers { permanent_uuid: "7628b292019d4388b096a4f96b2283eb" member_type: VOTER last_known_addr { host: "127.102.10.1" port: 36483 } } peers { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" member_type: VOTER last_known_addr { host: "127.102.10.2" port: 49416 } } peers { permanent_uuid: "5a9fdcf0227045cf9f868590152defd9" member_type: VOTER last_known_addr { host: "127.102.10.0" port: 41575 } } } W0225 21:50:42.496995 1711 raft_consensus_state.cc:524] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0225 21:50:42.516047 26122 master_failover-itest.cc:250] Pausing leader master I0225 21:50:42.989564 1480 raft_consensus.cc:831] T a4730e4d785a49069ba2bb382ede3f9e P 7628b292019d4388b096a4f96b2283eb [term 1 FOLLOWER]: Refusing update from remote peer 5a9fdcf0227045cf9f868590152defd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0225 21:50:42.989781 1601 raft_consensus.cc:831] T a4730e4d785a49069ba2bb382ede3f9e P abbe08b58a6b4e30b0250bd1288a9f32 [term 1 FOLLOWER]: Refusing update from remote peer 5a9fdcf0227045cf9f868590152defd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0225 21:50:42.991610 1712 consensus_queue.cc:572] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [LEADER]: Connected to new peer: Peer: abbe08b58a6b4e30b0250bd1288a9f32, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 21:50:42.991889 1706 consensus_queue.cc:572] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [LEADER]: Connected to new peer: Peer: 7628b292019d4388b096a4f96b2283eb, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 21:50:43.010790 1715 raft_consensus_state.cc:517] T a4730e4d785a49069ba2bb382ede3f9e P 5a9fdcf0227045cf9f868590152defd9 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0225 21:50:43.190038 1287 raft_consensus.cc:358] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 2 FOLLOWER]: Failure of leader 4c72ab184ce64d999dd3ebf9e3720cd4 detected. Triggering leader election I0225 21:50:43.190172 1287 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 2 FOLLOWER]: Advancing to term 3 I0225 21:50:43.191946 1287 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.542s I0225 21:50:43.192100 1287 raft_consensus.cc:378] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:43.194339 1287 leader_election.cc:222] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 3 election: Requesting vote from peer 5cbd1640a4934d2e9bbf6d3a52aeba70 I0225 21:50:43.194732 1287 leader_election.cc:222] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 3 election: Requesting vote from peer 4c72ab184ce64d999dd3ebf9e3720cd4 I0225 21:50:43.196138 1222 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 2 FOLLOWER]: Advancing to term 3 I0225 21:50:43.197844 1222 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.511s I0225 21:50:43.199264 1222 raft_consensus.cc:1649] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1972008325413a943c1ee4f140ee66 in term 3. I0225 21:50:43.199983 1154 leader_election.cc:346] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 3 election: Vote granted by peer 5cbd1640a4934d2e9bbf6d3a52aeba70 I0225 21:50:43.200134 1154 leader_election.cc:247] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0225 21:50:43.201022 1716 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.511s I0225 21:50:43.201274 1716 raft_consensus.cc:1831] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 3 FOLLOWER]: Leader election won for term 3 I0225 21:50:43.202087 1716 raft_consensus.cc:442] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 3 LEADER]: Becoming Leader. State: Replica: 7d1972008325413a943c1ee4f140ee66, State: 1, Role: LEADER Watermarks: {Received: term: 2 index: 8 Committed: term: 2 index: 8} I0225 21:50:43.202435 1716 consensus_queue.cc:145] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [LEADER]: Queue going to LEADER mode. State: All replicated op: 2.8, Majority replicated op: 2.8, Committed index: 2.8, Last appended: 2.8, Current term: 3, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 21:50:43.204773 1717 sys_catalog.cc:227] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1972008325413a943c1ee4f140ee66. Latest consensus state: current_term: 3 leader_uuid: "7d1972008325413a943c1ee4f140ee66" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:43.205152 1717 sys_catalog.cc:230] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER W0225 21:50:43.206110 1720 raft_consensus_state.cc:524] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 3 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 8, New majority replicated is: term: 2 index: 8, Current term is: 3 I0225 21:50:43.206511 1721 catalog_manager.cc:520] Loading table and tablet metadata into memory... I0225 21:50:43.211338 1721 catalog_manager.cc:202] Loaded metadata for table [id=93c0db77fc8447458439412b937c94d3] I0225 21:50:43.215600 1721 catalog_manager.cc:266] Loaded metadata for tablet a4730e4d785a49069ba2bb382ede3f9e (table testAlterTableSync [id=93c0db77fc8447458439412b937c94d3]) I0225 21:50:43.705875 1213 raft_consensus.cc:831] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [term 3 FOLLOWER]: Refusing update from remote peer 7d1972008325413a943c1ee4f140ee66: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0225 21:50:43.707128 1716 consensus_queue.cc:572] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [LEADER]: Connected to new peer: Peer: 5cbd1640a4934d2e9bbf6d3a52aeba70, Is new: false, Last received: 2.8, Next index: 9, Last known committed idx: 7, Last exchange result: ERROR, Needs remote bootstrap: false I0225 21:50:43.711757 1722 sys_catalog.cc:227] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1972008325413a943c1ee4f140ee66. Latest consensus state: current_term: 3 leader_uuid: "7d1972008325413a943c1ee4f140ee66" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:43.712179 1722 sys_catalog.cc:230] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 21:50:43.714573 1724 raft_consensus_state.cc:517] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [term 3 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 2 index: 8 New committed index is: term: 3 index: 9 I0225 21:50:43.717097 1722 sys_catalog.cc:227] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7d1972008325413a943c1ee4f140ee66" config { opid_index: -1 local: false peers { permanent_uuid: "7d1972008325413a943c1ee4f140ee66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "5cbd1640a4934d2e9bbf6d3a52aeba70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "4c72ab184ce64d999dd3ebf9e3720cd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 21:50:43.717396 1722 sys_catalog.cc:230] T 00000000000000000000000000000000 P 5cbd1640a4934d2e9bbf6d3a52aeba70 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER W0225 21:50:43.738008 1154 leader_election.cc:274] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4c72ab184ce64d999dd3ebf9e3720cd4: Timed out: RequestConsensusVote RPC to 127.0.0.1:11012 timed out after 0.542s W0225 21:50:44.705926 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 1 times. W0225 21:50:46.143729 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 2 times. W0225 21:50:47.516702 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 3 times. W0225 21:50:47.517638 26122 client-internal.cc:178] Unable to send the request (table { table_name: "testAlterTableSync" } new_table_name: "testAlterTableSyncRenamed") to leader Master (127.0.0.1:11012): Timed out: AlterTable RPC to 127.0.0.1:11012 timed out after 5.000s I0225 21:50:47.517909 26122 client-internal.cc:182] Determining the new leader Master and retrying... I0225 21:50:47.523695 1162 catalog_manager.cc:1083] Servicing AlterTable request from {real_user=adar, eff_user=} at 127.0.0.1:55718: table { table_name: "testAlterTableSync" } new_table_name: "testAlterTableSyncRenamed" W0225 21:50:47.542672 1162 catalog_manager.cc:1805] Unable to reset TS proxy: Not found: no leader W0225 21:50:47.542961 1162 catalog_manager.cc:2421] Failed to send alter table request: Not found: Failed to reset TS proxy: no leader I0225 21:50:47.545518 1163 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.549075 1165 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.552866 1164 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.557066 1166 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.561800 1167 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.567121 1158 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.573210 1159 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.580164 1160 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.588476 1161 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.598078 1162 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.609582 1163 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.623363 1165 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.640092 1164 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.660465 1166 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.685444 1167 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.716111 1158 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.753847 1159 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.800344 1160 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.858090 1161 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:47.929641 1162 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:48.018656 1163 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:48.129523 1165 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:48.267251 1164 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:48.439106 1166 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:48.653151 1167 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:48.920130 1158 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:50:49.017791 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 4 times. I0225 21:50:49.253319 1159 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:49.669209 1160 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:50.188400 1161 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:50:50.421080 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 5 times. I0225 21:50:50.837195 1162 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done I0225 21:50:51.647651 1163 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:50:51.922144 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 6 times. I0225 21:50:52.660140 1165 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:50:52.710206 1541 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11012: Timed out: Failed to send heartbeat: TSHeartbeat RPC to 127.0.0.1:11012 timed out after 10.000s I0225 21:50:52.714758 1541 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 21:50:52.717371 1167 catalog_manager.cc:1457] TS 7628b292019d4388b096a4f96b2283eb does not have the latest schema for tablet a4730e4d785a49069ba2bb382ede3f9e (table testAlterTableSyncRenamed [id=93c0db77fc8447458439412b937c94d3]). Expected version 1 got 0 W0225 21:50:52.719916 1663 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11012: Timed out: Failed to send heartbeat: TSHeartbeat RPC to 127.0.0.1:11012 timed out after 10.000s I0225 21:50:52.724457 1663 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 21:50:52.726284 1160 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" instance_seqno: 1456465841472939 } as {real_user=adar, eff_user=} at 127.102.10.2:51577; Asking this server to re-register. I0225 21:50:52.727195 1663 heartbeater.cc:328] Registering TS with master... I0225 21:50:52.727350 1663 heartbeater.cc:334] Sending a full tablet report to master... I0225 21:50:52.728979 1161 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "abbe08b58a6b4e30b0250bd1288a9f32" instance_seqno: 1456465841472939 } with Master W0225 21:50:52.729287 1167 catalog_manager.cc:1805] Unable to reset TS proxy: Not found: no leader W0225 21:50:52.729406 1167 catalog_manager.cc:2421] Failed to send alter table request: Not found: Failed to reset TS proxy: no leader I0225 21:50:52.729640 1161 catalog_manager.cc:1457] TS abbe08b58a6b4e30b0250bd1288a9f32 does not have the latest schema for tablet a4730e4d785a49069ba2bb382ede3f9e (table testAlterTableSyncRenamed [id=93c0db77fc8447458439412b937c94d3]). Expected version 1 got 0 W0225 21:50:52.738718 1161 catalog_manager.cc:1805] Unable to reset TS proxy: Not found: no leader W0225 21:50:52.738850 1161 catalog_manager.cc:2421] Failed to send alter table request: Not found: Failed to reset TS proxy: no leader W0225 21:50:53.009840 1411 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11012: Timed out: Failed to send heartbeat: TSHeartbeat RPC to 127.0.0.1:11012 timed out after 10.000s W0225 21:50:53.423193 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 7 times. I0225 21:50:53.514513 1411 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 21:50:53.924967 1160 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:50:54.923575 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 8 times. I0225 21:50:55.505578 1160 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:50:56.356504 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 9 times. I0225 21:50:57.480446 1162 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:50:57.857549 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 10 times. W0225 21:50:59.357710 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 11 times. I0225 21:50:59.483155 1164 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:00.858636 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 12 times. I0225 21:51:01.486088 1159 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:02.359643 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 13 times. I0225 21:51:03.488894 1161 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:03.860586 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 14 times. W0225 21:51:05.360780 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 15 times. I0225 21:51:05.491506 1165 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:06.861678 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 16 times. I0225 21:51:07.494051 1158 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:08.362673 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 17 times. I0225 21:51:09.496788 1167 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:09.862743 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 18 times. W0225 21:51:11.363921 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 19 times. I0225 21:51:11.499503 1163 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:12.863989 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 20 times. I0225 21:51:13.502172 1166 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:14.365099 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 21 times. I0225 21:51:15.504809 1160 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:15.866094 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 22 times. W0225 21:51:17.366297 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 23 times. I0225 21:51:17.507167 1162 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:18.867156 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 24 times. I0225 21:51:19.509871 1164 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:20.368324 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 25 times. I0225 21:51:21.512564 1159 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:21.868158 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 26 times. W0225 21:51:23.369129 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 27 times. I0225 21:51:23.515266 1161 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:24.870115 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 28 times. I0225 21:51:25.518093 1165 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:26.370272 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 29 times. I0225 21:51:27.520720 1158 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:27.871356 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 30 times. W0225 21:51:29.371382 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 31 times. I0225 21:51:29.523380 1167 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:30.872763 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 32 times. I0225 21:51:31.526197 1163 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:32.372742 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 33 times. I0225 21:51:33.528828 1166 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:33.873584 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 34 times. W0225 21:51:35.374861 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 35 times. I0225 21:51:35.531275 1160 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:36.875792 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 36 times. I0225 21:51:37.533746 1162 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:38.375922 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 37 times. I0225 21:51:39.536217 1164 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:39.876942 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 38 times. W0225 21:51:41.377213 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 39 times. I0225 21:51:41.538604 1159 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:42.878126 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 40 times. I0225 21:51:43.541236 1161 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:44.379171 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 41 times. I0225 21:51:45.543777 1165 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:45.879851 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 42 times. W0225 21:51:47.380115 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 43 times. I0225 21:51:47.546210 1158 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:48.880985 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 44 times. I0225 21:51:49.548822 1167 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:50.381167 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 45 times. I0225 21:51:51.551347 1163 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:51.881943 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 46 times. W0225 21:51:53.383204 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 47 times. I0225 21:51:53.553910 1166 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:54.883550 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 48 times. I0225 21:51:55.556378 1160 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:56.384580 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 49 times. I0225 21:51:57.558843 1162 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:51:57.885568 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 50 times. W0225 21:51:59.385947 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 51 times. I0225 21:51:59.561298 1164 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:52:00.886872 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 52 times. I0225 21:52:01.563796 1159 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:52:02.387079 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 53 times. I0225 21:52:03.566198 1161 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:52:03.887917 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 54 times. W0225 21:52:05.388241 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 55 times. I0225 21:52:05.568613 1165 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:52:06.889369 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 56 times. I0225 21:52:07.571233 1158 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:52:08.390357 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 57 times. I0225 21:52:09.573855 1167 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done W0225 21:52:09.890437 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 58 times. W0225 21:52:11.391547 1154 consensus_peers.cc:319] T 00000000000000000000000000000000 P 7d1972008325413a943c1ee4f140ee66 -> Peer 4c72ab184ce64d999dd3ebf9e3720cd4 (127.0.0.1:11012): Couldn't send request to peer 4c72ab184ce64d999dd3ebf9e3720cd4 for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11012 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 59 times. I0225 21:52:11.576328 1163 catalog_manager.cc:1232] Table table_name: "testAlterTableSyncRenamed" exists, alter is not done /home/adar/kudu/src/kudu/integration-tests/master_failover-itest.cc:255: Failure Failed Bad status: Timed out: Timed out waiting for AlterTable terminate called after throwing an instance of 'testing::internal::GoogleTestFailureException' what(): /home/adar/kudu/src/kudu/integration-tests/master_failover-itest.cc:255: Failure Failed Bad status: Timed out: Timed out waiting for AlterTable W0225 21:52:11.578896 1154 connection.cc:400] client connection to 127.0.0.1:11012 recv error: Network error: recv error: Connection reset by peer (error 104) *** Aborted at 1456465931 (unix time) try "date -d @1456465931" if you are using GNU date *** W0225 21:52:11.578925 1304 connection.cc:400] client connection to 127.0.0.1:11012 recv error: Network error: recv error: Connection reset by peer (error 104) PC: @ 0x3ae0a32625 (unknown) *** SIGABRT (@0x10ba0000660a) received by PID 26122 (TID 0x7fc93f7ee8e0) from PID 26122; stack trace: *** @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0a32625 (unknown) at ??:0 W0225 21:52:11.580318 1553 connection.cc:400] client connection to 127.0.0.1:11012 recv error: Network error: recv error: Connection reset by peer (error 104) W0225 21:52:11.580507 1434 connection.cc:400] client connection to 127.0.0.1:11012 recv error: Network error: recv error: Connection reset by peer (error 104) @ 0x3ae0a33e05 (unknown) at ??:0 W0225 21:52:11.580729 1673 connection.cc:400] client connection to 127.0.0.1:11012 recv error: Network error: recv error: Connection reset by peer (error 104) @ 0x3ae3abea7d (unknown) at ??:0 @ 0x3ae3abcbd6 (unknown) at ??:0 W0225 21:52:11.581321 1244 consensus_peers.cc:319] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 -> Peer 5cbd1640a4934d2e9bbf6d3a52aeba70 (127.0.0.1:11011): Couldn't send request to peer 5cbd1640a4934d2e9bbf6d3a52aeba70 for tablet 00000000000000000000000000000000 Status: Network error: Recv() got EOF from remote (error 108). Retrying in the next heartbeat period. Already tried 1 times. @ 0x3ae3abcc03 (unknown) at ??:0 W0225 21:52:11.581774 1244 consensus_peers.cc:319] T 00000000000000000000000000000000 P 4c72ab184ce64d999dd3ebf9e3720cd4 -> Peer 7d1972008325413a943c1ee4f140ee66 (127.0.0.1:11010): Couldn't send request to peer 7d1972008325413a943c1ee4f140ee66 for tablet 00000000000000000000000000000000 Status: Network error: Recv() got EOF from remote (error 108). Retrying in the next heartbeat period. Already tried 1 times. @ 0x3ae3abcc86 (unknown) at ??:0 @ 0x7fc942913172 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0 @ 0x7fc9429077dd testing::UnitTest::Run() at ??:0 @ 0x7fc950877b90 RUN_ALL_TESTS() at ??:0 @ 0x7fc950875d9b main at ??:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x428df5 (unknown) at ??:0 *** Aborted at 1456465931 (unix time) try "date -d @1456465931" if you are using GNU date *** *** Aborted at 1456465931 (unix time) try "date -d @1456465931" if you are using GNU date *** *** Aborted at 1456465931 (unix time) try "date -d @1456465931" if you are using GNU date *** *** Aborted at 1456465931 (unix time) try "date -d @1456465931" if you are using GNU date *** *** Aborted at 1456465931 (unix time) try "date -d @1456465931" if you are using GNU date *** *** Aborted at 1456465931 (unix time) try "date -d @1456465931" if you are using GNU date *** PC: @ 0x3ae0e0ef3d (unknown) PC: @ 0x3ae0e0ef3d (unknown) PC: @ 0x3ae0e0ef3d (unknown) PC: @ 0x3ae0e0ef3d (unknown) PC: @ 0x3ae0e0ef3d (unknown) *** SIGTERM (@0x10ba0000660a) received by PID 1149 (TID 0x7f2415508820) from PID 26122; stack trace: *** PC: @ 0x3ae0e0ef3d (unknown) *** SIGTERM (@0x10ba0000660a) received by PID 1282 (TID 0x7f2eb729c820) from PID 26122; stack trace: *** *** SIGTERM (@0x10ba0000660a) received by PID 1417 (TID 0x7fd18930e820) from PID 26122; stack trace: *** *** SIGTERM (@0x10ba0000660a) received by PID 1236 (TID 0x7fec7f5e4820) from PID 26122; stack trace: *** *** SIGTERM (@0x10ba0000660a) received by PID 1191 (TID 0x7fd742eda820) from PID 26122; stack trace: *** *** SIGTERM (@0x10ba0000660a) received by PID 1547 (TID 0x7f5188213820) from PID 26122; stack trace: *** @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x7f24176aa1c8 base::SleepForNanoseconds() at ??:0 @ 0x7f2eb8b671c8 base::SleepForNanoseconds() at ??:0 @ 0x7fd18abd91c8 base::SleepForNanoseconds() at ??:0 @ 0x7fd74507c1c8 base::SleepForNanoseconds() at ??:0 @ 0x7fec817861c8 base::SleepForNanoseconds() at ??:0 @ 0x7f5189ade1c8 base::SleepForNanoseconds() at ??:0 @ 0x7f24189d553e kudu::SleepFor() at ??:0 @ 0x4ea327 kudu::master::MasterMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:287 @ 0x4e7f3e main at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:311 @ 0x7f2eb9e9153e kudu::SleepFor() at ??:0 @ 0x4ea36e kudu::tserver::TabletServerMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:503 @ 0x4e7f5e main at crtstuff.c:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x41dd45 (unknown) at ??:0 @ 0x41dd65 (unknown) at ??:0 @ 0x7f518ae0853e kudu::SleepFor() at ??:0 @ 0x7fd7463a753e kudu::SleepFor() at ??:0 @ 0x4ea327 kudu::master::MasterMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:287 @ 0x4ea36e kudu::tserver::TabletServerMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:503 @ 0x4e7f3e main at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:311 @ 0x4e7f5e main at crtstuff.c:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x7fd18bf0353e kudu::SleepFor() at ??:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x7fec82ab153e kudu::SleepFor() at ??:0 @ 0x4ea36e kudu::tserver::TabletServerMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:503 @ 0x4ea327 kudu::master::MasterMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:287 @ 0x4e7f5e main at crtstuff.c:0 @ 0x41dd45 (unknown) at ??:0 @ 0x4e7f3e main at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:311 @ 0x41dd65 (unknown) at ??:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x41dd65 (unknown) at ??:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x41dd45 (unknown) at ??:0