[ RUN ] MasterFailoverTest.TestRenameTableSync I0225 14:50:46.900344 27958 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-0 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-0 --rpc_bind_addresses=127.0.0.1:11010 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-0 --leader_failure_max_missed_heartbeat_periods=1.0 --master_addresses=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --enable_leader_failure_detection=true --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 14:50:47.233867 18071 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 14:50:47.234663 18071 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 14:50:47.244170 18071 master_main.cc:58] Initializing master server... I0225 14:50:47.245123 18071 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 934896 I0225 14:50:47.250963 18071 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-0/instance: No such file or directory (error 2) I0225 14:50:47.251055 18071 server_base.cc:165] Creating new FS layout I0225 14:50:47.267750 18071 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-0/instance: uuid: "3e305ed53fac466ba564ca1e7586e84a" format_stamp: "Formatted at 2016-02-25 22:50:47 on ve0518.halxg.cloudera.com" I0225 14:50:47.279964 18071 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-0 uuid: "3e305ed53fac466ba564ca1e7586e84a" format_stamp: "Formatted at 2016-02-25 22:50:47 on ve0518.halxg.cloudera.com" I0225 14:50:47.306082 18071 master_main.cc:61] Starting Master server... I0225 14:50:47.346119 18071 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:11010 I0225 14:50:47.347170 18071 webserver.cc:135] Starting webserver on localhost:0 I0225 14:50:47.347224 18071 webserver.cc:146] Document root disabled I0225 14:50:47.348369 18071 net_util.cc:175] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0225 14:50:47.350095 18071 webserver.cc:229] Webserver started. Bound to: http://127.0.0.1:41592/ I0225 14:50:47.352422 18071 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-0/info.pb I0225 14:50:47.359802 27958 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-master as pid 18071 I0225 14:50:47.360307 27958 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-1 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-1 --rpc_bind_addresses=127.0.0.1:11011 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-1 --leader_failure_max_missed_heartbeat_periods=1.0 --master_addresses=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --enable_leader_failure_detection=true --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 14:50:47.374673 18112 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } has no permanent_uuid. Determining permanent_uuid... W0225 14:50:47.378059 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11010, user_credentials={real_user=adar, eff_user=}} I0225 14:50:47.400357 18112 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } has no permanent_uuid. Determining permanent_uuid... W0225 14:50:47.400806 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 14:50:47.402771 18112 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 14:50:47.463212 18112 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 1 W0225 14:50:47.463634 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 14:50:47.465044 18112 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 14:50:47.516257 18112 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 2 W0225 14:50:47.516623 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 14:50:47.517812 18112 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 14:50:47.620050 18112 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 3 W0225 14:50:47.620412 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 14:50:47.621814 18112 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 14:50:47.692313 18113 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 14:50:47.693225 18113 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 14:50:47.703357 18113 master_main.cc:58] Initializing master server... I0225 14:50:47.704324 18113 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 934896 I0225 14:50:47.710350 18113 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-1/instance: No such file or directory (error 2) I0225 14:50:47.710455 18113 server_base.cc:165] Creating new FS layout I0225 14:50:47.719384 18113 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-1/instance: uuid: "af53ee3fc638416993de9457afc302c5" format_stamp: "Formatted at 2016-02-25 22:50:47 on ve0518.halxg.cloudera.com" I0225 14:50:47.732414 18113 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-1 uuid: "af53ee3fc638416993de9457afc302c5" format_stamp: "Formatted at 2016-02-25 22:50:47 on ve0518.halxg.cloudera.com" I0225 14:50:47.751032 18112 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 4 W0225 14:50:47.751360 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} W0225 14:50:47.752558 18112 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11011: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11011: connect: Connection refused (error 111) I0225 14:50:47.757042 18113 master_main.cc:61] Starting Master server... I0225 14:50:47.799902 18113 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:11011 I0225 14:50:47.800881 18113 webserver.cc:135] Starting webserver on localhost:0 I0225 14:50:47.800931 18113 webserver.cc:146] Document root disabled I0225 14:50:47.802032 18113 net_util.cc:175] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0225 14:50:47.803731 18113 webserver.cc:229] Webserver started. Bound to: http://127.0.0.1:39738/ I0225 14:50:47.805397 18113 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-1/info.pb I0225 14:50:47.811048 27958 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-master as pid 18113 I0225 14:50:47.811581 27958 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-2 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-2 --rpc_bind_addresses=127.0.0.1:11012 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-2 --leader_failure_max_missed_heartbeat_periods=1.0 --master_addresses=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --enable_leader_failure_detection=true --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 14:50:47.826354 18157 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } has no permanent_uuid. Determining permanent_uuid... W0225 14:50:47.829653 18121 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11010, user_credentials={real_user=adar, eff_user=}} I0225 14:50:47.843147 18157 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } has no permanent_uuid. Determining permanent_uuid... W0225 14:50:47.843569 18121 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} I0225 14:50:47.854439 18157 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } has no permanent_uuid. Determining permanent_uuid... W0225 14:50:47.854948 18121 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 14:50:47.856894 18157 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 14:50:47.917342 18157 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 1 W0225 14:50:47.917783 18121 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 14:50:47.919150 18157 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 14:50:47.970409 18157 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 2 W0225 14:50:47.970779 18121 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 14:50:47.972043 18157 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 14:50:48.034853 18112 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } attempt: 5 W0225 14:50:48.035298 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} I0225 14:50:48.039424 18112 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } has no permanent_uuid. Determining permanent_uuid... W0225 14:50:48.039913 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 14:50:48.040787 18112 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 14:50:48.074280 18157 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 3 W0225 14:50:48.074676 18121 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 14:50:48.075976 18157 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 14:50:48.089993 18112 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 1 W0225 14:50:48.090409 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 14:50:48.091511 18112 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 14:50:48.142752 18112 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 2 W0225 14:50:48.143224 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 14:50:48.144285 18112 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 14:50:48.156507 18158 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 14:50:48.157443 18158 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 14:50:48.169255 18158 master_main.cc:58] Initializing master server... I0225 14:50:48.170397 18158 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 935396 I0225 14:50:48.177702 18158 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-2/instance: No such file or directory (error 2) I0225 14:50:48.177829 18158 server_base.cc:165] Creating new FS layout I0225 14:50:48.186868 18158 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-2/instance: uuid: "f9bc2ec9795e45f99414d3460b8af958" format_stamp: "Formatted at 2016-02-25 22:50:48 on ve0518.halxg.cloudera.com" I0225 14:50:48.199940 18158 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-2 uuid: "f9bc2ec9795e45f99414d3460b8af958" format_stamp: "Formatted at 2016-02-25 22:50:48 on ve0518.halxg.cloudera.com" I0225 14:50:48.205202 18157 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 4 W0225 14:50:48.205565 18121 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} W0225 14:50:48.206701 18157 consensus_peers.cc:450] Error getting permanent uuid from config peer 127.0.0.1:11012: Network error: Client connection negotiation failed: client connection to 127.0.0.1:11012: connect: Connection refused (error 111) I0225 14:50:48.222259 18158 master_main.cc:61] Starting Master server... I0225 14:50:48.236975 18158 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:11012 I0225 14:50:48.237802 18158 webserver.cc:135] Starting webserver on localhost:0 I0225 14:50:48.237841 18158 webserver.cc:146] Document root disabled I0225 14:50:48.238559 18158 net_util.cc:175] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0225 14:50:48.239382 18158 webserver.cc:229] Webserver started. Bound to: http://127.0.0.1:60490/ I0225 14:50:48.240442 18158 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/master-2/info.pb I0225 14:50:48.241461 18112 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 3 W0225 14:50:48.241839 18076 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} I0225 14:50:48.243330 27958 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-master as pid 18158 I0225 14:50:48.244048 27958 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-0 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-0 --rpc_bind_addresses=127.109.54.0:0 --local_ip_for_outbound_sockets=127.109.54.0 --webserver_interface=127.109.54.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-0 --heartbeat_rpc_timeout_ms=1000 --heartbeat_max_failures_before_backoff=1 --heartbeat_interval_ms=500 --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 14:50:48.258622 18202 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } has no permanent_uuid. Determining permanent_uuid... I0225 14:50:48.260283 18112 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a: Bootstrap starting. W0225 14:50:48.261477 18166 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11010, user_credentials={real_user=adar, eff_user=}} I0225 14:50:48.264873 18112 tablet_bootstrap.cc:524] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a: Time spent opening tablet: real 0.001s user 0.002s sys 0.000s I0225 14:50:48.265374 18112 tablet_bootstrap.cc:464] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a: No blocks or log segments found. Creating new log. I0225 14:50:48.267079 18112 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 14:50:48.269148 18202 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } has no permanent_uuid. Determining permanent_uuid... W0225 14:50:48.269570 18166 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11011, user_credentials={real_user=adar, eff_user=}} I0225 14:50:48.270787 18112 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a: No bootstrap required, opened a new log I0225 14:50:48.273089 18202 sys_catalog.cc:196] member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } has no permanent_uuid. Determining permanent_uuid... W0225 14:50:48.273550 18166 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} I0225 14:50:48.276969 18112 raft_consensus.cc:268] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:50:48.278558 18112 raft_consensus.cc:300] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 14:50:48.278873 18112 raft_consensus.cc:477] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e305ed53fac466ba564ca1e7586e84a, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 14:50:48.279211 18112 consensus_queue.cc:162] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 14:50:48.280174 18112 sys_catalog.cc:227] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:48.280472 18112 sys_catalog.cc:230] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:48.280246 18210 sys_catalog.cc:227] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:48.280591 18210 sys_catalog.cc:230] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:48.282711 18112 sys_catalog.cc:299] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [sys.catalog]: configured and running, proceeding with master startup. I0225 14:50:48.283464 18071 master_main.cc:64] Master server successfully started. I0225 14:50:48.283377 18202 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958: Bootstrap starting. I0225 14:50:48.288065 18202 tablet_bootstrap.cc:524] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0225 14:50:48.288599 18202 tablet_bootstrap.cc:464] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958: No blocks or log segments found. Creating new log. I0225 14:50:48.290413 18202 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 14:50:48.294034 18202 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958: No bootstrap required, opened a new log I0225 14:50:48.300420 18202 raft_consensus.cc:268] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:50:48.302047 18202 raft_consensus.cc:300] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 14:50:48.302364 18202 raft_consensus.cc:477] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bc2ec9795e45f99414d3460b8af958, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 14:50:48.302772 18202 consensus_queue.cc:162] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 14:50:48.303675 18202 sys_catalog.cc:227] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:48.303952 18202 sys_catalog.cc:230] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:48.303957 18215 sys_catalog.cc:227] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:48.304384 18215 sys_catalog.cc:230] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:48.305461 18202 sys_catalog.cc:299] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: configured and running, proceeding with master startup. I0225 14:50:48.306103 18158 master_main.cc:64] Master server successfully started. I0225 14:50:48.488952 18157 consensus_peers.cc:460] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } attempt: 5 W0225 14:50:48.489312 18121 reactor.cc:227] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:11012, user_credentials={real_user=adar, eff_user=}} I0225 14:50:48.497159 18157 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5: Bootstrap starting. I0225 14:50:48.500010 18157 tablet_bootstrap.cc:524] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0225 14:50:48.500329 18157 tablet_bootstrap.cc:464] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5: No blocks or log segments found. Creating new log. I0225 14:50:48.501258 18157 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 14:50:48.504976 18157 tablet_bootstrap.cc:363] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5: No bootstrap required, opened a new log I0225 14:50:48.511200 18157 raft_consensus.cc:268] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:50:48.512812 18157 raft_consensus.cc:300] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 14:50:48.513103 18157 raft_consensus.cc:477] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af53ee3fc638416993de9457afc302c5, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 14:50:48.513067 18204 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 14:50:48.513357 18204 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 14:50:48.513485 18157 consensus_queue.cc:162] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 14:50:48.514397 18157 sys_catalog.cc:227] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:48.514590 18157 sys_catalog.cc:230] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:48.514617 18220 sys_catalog.cc:227] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:48.515009 18220 sys_catalog.cc:230] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:48.515969 18157 sys_catalog.cc:299] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: configured and running, proceeding with master startup. I0225 14:50:48.516677 18113 master_main.cc:64] Master server successfully started. I0225 14:50:48.520771 18204 tablet_server_main.cc:54] Initializing tablet server... I0225 14:50:48.522869 18204 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 935396 I0225 14:50:48.526619 18204 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-0/instance: No such file or directory (error 2) I0225 14:50:48.526701 18204 server_base.cc:165] Creating new FS layout I0225 14:50:48.533690 18204 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-0/instance: uuid: "713690c2ff104bd6a70fbfa8303fbe61" format_stamp: "Formatted at 2016-02-25 22:50:48 on ve0518.halxg.cloudera.com" I0225 14:50:48.546452 18204 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-0 uuid: "713690c2ff104bd6a70fbfa8303fbe61" format_stamp: "Formatted at 2016-02-25 22:50:48 on ve0518.halxg.cloudera.com" I0225 14:50:48.568605 18204 tablet_server_main.cc:57] Starting tablet server... I0225 14:50:48.602779 18204 rpc_server.cc:164] RPC server started. Bound to: 127.109.54.0:33864 I0225 14:50:48.603749 18204 webserver.cc:135] Starting webserver on 127.109.54.0:0 I0225 14:50:48.603811 18204 webserver.cc:146] Document root disabled I0225 14:50:48.604996 18204 webserver.cc:229] Webserver started. Bound to: http://127.109.54.0:53669/ I0225 14:50:48.606194 18204 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-0/info.pb I0225 14:50:48.607379 18204 tablet_server_main.cc:60] Tablet server successfully started. I0225 14:50:48.612149 27958 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-tserver as pid 18204 I0225 14:50:48.612867 27958 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-1 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-1 --rpc_bind_addresses=127.109.54.1:0 --local_ip_for_outbound_sockets=127.109.54.1 --webserver_interface=127.109.54.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-1 --heartbeat_rpc_timeout_ms=1000 --heartbeat_max_failures_before_backoff=1 --heartbeat_interval_ms=500 --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-1/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 14:50:48.622298 18209 raft_consensus.cc:362] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0225 14:50:48.622505 18209 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 0 FOLLOWER]: Advancing to term 1 I0225 14:50:48.624332 18209 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.531s I0225 14:50:48.624467 18209 raft_consensus.cc:378] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:50:48.627936 18209 leader_election.cc:222] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [CANDIDATE]: Term 1 election: Requesting vote from peer af53ee3fc638416993de9457afc302c5 I0225 14:50:48.628510 18209 leader_election.cc:222] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [CANDIDATE]: Term 1 election: Requesting vote from peer f9bc2ec9795e45f99414d3460b8af958 I0225 14:50:48.630309 18138 raft_consensus.cc:1988] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 0 FOLLOWER]: Advancing to term 1 I0225 14:50:48.630384 18183 raft_consensus.cc:1988] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 0 FOLLOWER]: Advancing to term 1 I0225 14:50:48.632311 18138 raft_consensus.cc:1939] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.517s I0225 14:50:48.632624 18183 raft_consensus.cc:1939] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.547s I0225 14:50:48.633790 18138 raft_consensus.cc:1649] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e305ed53fac466ba564ca1e7586e84a in term 1. I0225 14:50:48.633949 18183 raft_consensus.cc:1649] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e305ed53fac466ba564ca1e7586e84a in term 1. I0225 14:50:48.634693 18076 leader_election.cc:346] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [CANDIDATE]: Term 1 election: Vote granted by peer af53ee3fc638416993de9457afc302c5 I0225 14:50:48.634877 18076 leader_election.cc:247] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0225 14:50:48.635395 18076 leader_election.cc:346] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [CANDIDATE]: Term 1 election: Vote granted by peer f9bc2ec9795e45f99414d3460b8af958 I0225 14:50:48.635393 18210 raft_consensus.cc:1939] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.517s I0225 14:50:48.635722 18210 raft_consensus.cc:1831] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 FOLLOWER]: Leader election won for term 1 I0225 14:50:48.636800 18210 raft_consensus.cc:442] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 LEADER]: Becoming Leader. State: Replica: 3e305ed53fac466ba564ca1e7586e84a, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 14:50:48.637586 18210 consensus_queue.cc:145] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:50:48.647150 18340 sys_catalog.cc:227] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e305ed53fac466ba564ca1e7586e84a. Latest consensus state: current_term: 1 leader_uuid: "3e305ed53fac466ba564ca1e7586e84a" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:48.647589 18340 sys_catalog.cc:230] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER I0225 14:50:48.649224 18343 catalog_manager.cc:520] Loading table and tablet metadata into memory... I0225 14:50:48.649657 18333 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 14:50:48.649893 18333 heartbeater.cc:328] Registering TS with master... I0225 14:50:48.650066 18333 heartbeater.cc:334] Sending a full tablet report to master... W0225 14:50:48.650471 18344 raft_consensus_state.cc:524] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 W0225 14:50:48.651444 18086 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Service unavailable: Leader not yet ready to serve requests W0225 14:50:48.652184 18333 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11010: Service unavailable: master is no longer the leader W0225 14:50:48.652251 18333 heartbeater.cc:287] Failed 1 heartbeats in a row: no longer allowing fast heartbeat attempts. I0225 14:50:48.655786 18333 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 14:50:48.655860 18333 heartbeater.cc:328] Registering TS with master... I0225 14:50:48.655948 18333 heartbeater.cc:334] Sending a full tablet report to master... W0225 14:50:48.656973 18089 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Service unavailable: Leader not yet ready to serve requests W0225 14:50:48.657564 18333 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11010: Service unavailable: master is no longer the leader I0225 14:50:48.898996 18339 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 14:50:48.899312 18339 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 14:50:48.906615 18339 tablet_server_main.cc:54] Initializing tablet server... I0225 14:50:48.908815 18339 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 935396 I0225 14:50:48.912642 18339 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-1/instance: No such file or directory (error 2) I0225 14:50:48.912749 18339 server_base.cc:165] Creating new FS layout I0225 14:50:48.919929 18339 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-1/instance: uuid: "c30b588e10004dcdb01341ed61765099" format_stamp: "Formatted at 2016-02-25 22:50:48 on ve0518.halxg.cloudera.com" I0225 14:50:48.932456 18339 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-1 uuid: "c30b588e10004dcdb01341ed61765099" format_stamp: "Formatted at 2016-02-25 22:50:48 on ve0518.halxg.cloudera.com" I0225 14:50:48.951527 18339 tablet_server_main.cc:57] Starting tablet server... I0225 14:50:48.983415 18339 rpc_server.cc:164] RPC server started. Bound to: 127.109.54.1:44997 I0225 14:50:48.984241 18339 webserver.cc:135] Starting webserver on 127.109.54.1:0 I0225 14:50:48.984289 18339 webserver.cc:146] Document root disabled I0225 14:50:48.985301 18339 webserver.cc:229] Webserver started. Bound to: http://127.109.54.1:35485/ I0225 14:50:48.986405 18339 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-1/info.pb I0225 14:50:48.987471 18339 tablet_server_main.cc:60] Tablet server successfully started. I0225 14:50:48.993098 27958 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-tserver as pid 18339 I0225 14:50:48.993728 27958 external_mini_cluster.cc:546] Running /home/adar/kudu/build/asan/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-2 --fs_data_dirs=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-2 --rpc_bind_addresses=127.109.54.2:0 --local_ip_for_outbound_sockets=127.109.54.2 --webserver_interface=127.109.54.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:11010,127.0.0.1:11011,127.0.0.1:11012 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-2 --heartbeat_rpc_timeout_ms=1000 --heartbeat_max_failures_before_backoff=1 --heartbeat_interval_ms=500 --server_dump_info_path=/tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-2/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --logtostderr --logbuflevel=-1 I0225 14:50:49.001199 18457 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 14:50:49.001514 18457 heartbeater.cc:328] Registering TS with master... I0225 14:50:49.001790 18457 heartbeater.cc:334] Sending a full tablet report to master... I0225 14:50:49.003512 18082 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "c30b588e10004dcdb01341ed61765099" instance_seqno: 1456440648977430 } with Master I0225 14:50:49.143570 18139 raft_consensus.cc:831] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 1 FOLLOWER]: Refusing update from remote peer 3e305ed53fac466ba564ca1e7586e84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0225 14:50:49.143802 18184 raft_consensus.cc:831] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 1 FOLLOWER]: Refusing update from remote peer 3e305ed53fac466ba564ca1e7586e84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0225 14:50:49.145714 18210 consensus_queue.cc:572] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [LEADER]: Connected to new peer: Peer: af53ee3fc638416993de9457afc302c5, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 14:50:49.145977 18340 consensus_queue.cc:572] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [LEADER]: Connected to new peer: Peer: f9bc2ec9795e45f99414d3460b8af958, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 14:50:49.156102 18465 sys_catalog.cc:227] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e305ed53fac466ba564ca1e7586e84a. Latest consensus state: current_term: 1 leader_uuid: "3e305ed53fac466ba564ca1e7586e84a" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:49.156442 18465 sys_catalog.cc:230] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:49.160413 18464 sys_catalog.cc:227] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e305ed53fac466ba564ca1e7586e84a. Latest consensus state: current_term: 1 leader_uuid: "3e305ed53fac466ba564ca1e7586e84a" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:49.160946 18464 sys_catalog.cc:230] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:49.161336 18466 raft_consensus_state.cc:517] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0225 14:50:49.162603 18333 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 14:50:49.162780 18333 heartbeater.cc:328] Registering TS with master... I0225 14:50:49.162989 18333 heartbeater.cc:334] Sending a full tablet report to master... I0225 14:50:49.163980 18085 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" instance_seqno: 1456440648596239 } with Master I0225 14:50:49.164479 18465 sys_catalog.cc:227] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e305ed53fac466ba564ca1e7586e84a" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:49.164764 18465 sys_catalog.cc:230] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:49.166574 18464 sys_catalog.cc:227] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e305ed53fac466ba564ca1e7586e84a" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:49.166909 18464 sys_catalog.cc:230] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:49.311900 18463 mem_tracker.cc:137] MemTracker: hard memory limit is 75.496170 GB I0225 14:50:49.312233 18463 mem_tracker.cc:139] MemTracker: soft memory limit is 45.297703 GB I0225 14:50:49.321192 18463 tablet_server_main.cc:54] Initializing tablet server... I0225 14:50:49.323751 18463 hybrid_clock.cc:178] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 935896 I0225 14:50:49.328498 18463 server_base.cc:164] Could not load existing FS layout: Not found: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-2/instance: No such file or directory (error 2) I0225 14:50:49.328613 18463 server_base.cc:165] Creating new FS layout I0225 14:50:49.336992 18463 fs_manager.cc:345] Generated new instance metadata in path /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-2/instance: uuid: "0d18db0d3e6e410a9310469fb6927391" format_stamp: "Formatted at 2016-02-25 22:50:49 on ve0518.halxg.cloudera.com" I0225 14:50:49.350350 18463 fs_manager.cc:242] Opened local filesystem: /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-2 uuid: "0d18db0d3e6e410a9310469fb6927391" format_stamp: "Formatted at 2016-02-25 22:50:49 on ve0518.halxg.cloudera.com" I0225 14:50:49.371366 18463 tablet_server_main.cc:57] Starting tablet server... I0225 14:50:49.408696 18463 rpc_server.cc:164] RPC server started. Bound to: 127.109.54.2:59678 I0225 14:50:49.409647 18463 webserver.cc:135] Starting webserver on 127.109.54.2:0 I0225 14:50:49.409711 18463 webserver.cc:146] Document root disabled I0225 14:50:49.410923 18463 webserver.cc:229] Webserver started. Bound to: http://127.109.54.2:56140/ I0225 14:50:49.412127 18463 server_base.cc:234] Dumped server information to /tmp/kudutest-4282/master_failover-itest.MasterFailoverTest.TestRenameTableSync.1456437842167089-27958/minicluster-data/ts-2/info.pb I0225 14:50:49.412570 27958 external_mini_cluster.cc:582] Started /home/adar/kudu/build/asan/bin/kudu-tserver as pid 18463 I0225 14:50:49.413338 18463 tablet_server_main.cc:60] Tablet server successfully started. I0225 14:50:49.431352 18578 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11010 I0225 14:50:49.431706 18578 heartbeater.cc:328] Registering TS with master... I0225 14:50:49.431965 18578 heartbeater.cc:334] Sending a full tablet report to master... I0225 14:50:49.433498 18080 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" instance_seqno: 1456440649401216 } with Master I0225 14:50:49.441040 27958 external_mini_cluster.cc:318] 3 TS(s) registered with Master I0225 14:50:49.455847 18084 catalog_manager.cc:678] CreateTable from {real_user=adar, eff_user=} at 127.0.0.1:42062: name: "testAlterTableSync" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } split_rows { } partition_schema { } I0225 14:50:49.521358 18496 tablet_service.cc:602] Processing CreateTablet for tablet 5057465b8ee44d26ad2df63c369fbff3 (table=testAlterTableSync [id=07c5ef70986145e9ab072daea8b0cf20]), partition=range: [(), ()) I0225 14:50:49.521340 18251 tablet_service.cc:602] Processing CreateTablet for tablet 5057465b8ee44d26ad2df63c369fbff3 (table=testAlterTableSync [id=07c5ef70986145e9ab072daea8b0cf20]), partition=range: [(), ()) I0225 14:50:49.522204 18375 tablet_service.cc:602] Processing CreateTablet for tablet 5057465b8ee44d26ad2df63c369fbff3 (table=testAlterTableSync [id=07c5ef70986145e9ab072daea8b0cf20]), partition=range: [(), ()) I0225 14:50:49.536790 18375 ts_tablet_manager.cc:744] Registered tablet 5057465b8ee44d26ad2df63c369fbff3 I0225 14:50:49.537545 18496 ts_tablet_manager.cc:744] Registered tablet 5057465b8ee44d26ad2df63c369fbff3 I0225 14:50:49.537798 18603 ts_tablet_manager.cc:616] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099: Bootstrapping tablet I0225 14:50:49.538239 18251 ts_tablet_manager.cc:744] Registered tablet 5057465b8ee44d26ad2df63c369fbff3 I0225 14:50:49.538581 18604 ts_tablet_manager.cc:616] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391: Bootstrapping tablet I0225 14:50:49.539075 18606 ts_tablet_manager.cc:616] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61: Bootstrapping tablet I0225 14:50:49.540776 18606 tablet_bootstrap.cc:363] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61: Bootstrap starting. I0225 14:50:49.540750 18603 tablet_bootstrap.cc:363] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099: Bootstrap starting. I0225 14:50:49.541247 18604 tablet_bootstrap.cc:363] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391: Bootstrap starting. I0225 14:50:49.544137 18606 tablet_bootstrap.cc:524] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61: Time spent opening tablet: real 0.001s user 0.000s sys 0.001s I0225 14:50:49.544534 18606 tablet_bootstrap.cc:464] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61: No blocks or log segments found. Creating new log. I0225 14:50:49.545424 18603 tablet_bootstrap.cc:524] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0225 14:50:49.545452 18604 tablet_bootstrap.cc:524] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391: Time spent opening tablet: real 0.001s user 0.001s sys 0.000s I0225 14:50:49.545807 18604 tablet_bootstrap.cc:464] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391: No blocks or log segments found. Creating new log. I0225 14:50:49.545809 18603 tablet_bootstrap.cc:464] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099: No blocks or log segments found. Creating new log. I0225 14:50:49.545820 18606 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 14:50:49.547039 18604 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 14:50:49.547042 18603 log.cc:332] Log is configured to *not* fsync() on all Append() calls I0225 14:50:49.549036 18606 tablet_bootstrap.cc:363] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61: No bootstrap required, opened a new log I0225 14:50:49.549371 18606 ts_tablet_manager.cc:621] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s I0225 14:50:49.550282 18604 tablet_bootstrap.cc:363] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391: No bootstrap required, opened a new log I0225 14:50:49.550304 18603 tablet_bootstrap.cc:363] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099: No bootstrap required, opened a new log I0225 14:50:49.550631 18603 ts_tablet_manager.cc:621] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s I0225 14:50:49.550645 18604 ts_tablet_manager.cc:621] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s I0225 14:50:49.553658 18606 raft_consensus.cc:268] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" member_type: VOTER last_known_addr { host: "127.109.54.0" port: 33864 } } peers { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" member_type: VOTER last_known_addr { host: "127.109.54.2" port: 59678 } } peers { permanent_uuid: "c30b588e10004dcdb01341ed61765099" member_type: VOTER last_known_addr { host: "127.109.54.1" port: 44997 } } I0225 14:50:49.555071 18606 raft_consensus.cc:300] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 14:50:49.555299 18606 raft_consensus.cc:477] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713690c2ff104bd6a70fbfa8303fbe61, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 14:50:49.554937 18604 raft_consensus.cc:268] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" member_type: VOTER last_known_addr { host: "127.109.54.0" port: 33864 } } peers { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" member_type: VOTER last_known_addr { host: "127.109.54.2" port: 59678 } } peers { permanent_uuid: "c30b588e10004dcdb01341ed61765099" member_type: VOTER last_known_addr { host: "127.109.54.1" port: 44997 } } I0225 14:50:49.554941 18603 raft_consensus.cc:268] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" member_type: VOTER last_known_addr { host: "127.109.54.0" port: 33864 } } peers { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" member_type: VOTER last_known_addr { host: "127.109.54.2" port: 59678 } } peers { permanent_uuid: "c30b588e10004dcdb01341ed61765099" member_type: VOTER last_known_addr { host: "127.109.54.1" port: 44997 } } I0225 14:50:49.555541 18606 consensus_queue.cc:162] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 14:50:49.556407 18603 raft_consensus.cc:300] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 14:50:49.556406 18604 raft_consensus.cc:300] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0225 14:50:49.556617 18603 raft_consensus.cc:477] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30b588e10004dcdb01341ed61765099, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 14:50:49.556617 18604 raft_consensus.cc:477] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d18db0d3e6e410a9310469fb6927391, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 14:50:49.556888 18603 consensus_queue.cc:162] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 14:50:49.556891 18604 consensus_queue.cc:162] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0225 14:50:49.557271 18606 ts_tablet_manager.cc:643] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s I0225 14:50:49.558663 18604 ts_tablet_manager.cc:643] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s I0225 14:50:49.558908 18603 ts_tablet_manager.cc:643] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s I0225 14:50:50.005580 18613 raft_consensus.cc:362] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 0 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0225 14:50:50.005800 18613 raft_consensus.cc:1988] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 0 FOLLOWER]: Advancing to term 1 I0225 14:50:50.007899 18613 raft_consensus.cc:1939] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.633s I0225 14:50:50.008082 18613 raft_consensus.cc:378] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" member_type: VOTER last_known_addr { host: "127.109.54.0" port: 33864 } } peers { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" member_type: VOTER last_known_addr { host: "127.109.54.2" port: 59678 } } peers { permanent_uuid: "c30b588e10004dcdb01341ed61765099" member_type: VOTER last_known_addr { host: "127.109.54.1" port: 44997 } } I0225 14:50:50.012079 18613 leader_election.cc:222] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [CANDIDATE]: Term 1 election: Requesting vote from peer 0d18db0d3e6e410a9310469fb6927391 I0225 14:50:50.012758 18613 leader_election.cc:222] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [CANDIDATE]: Term 1 election: Requesting vote from peer c30b588e10004dcdb01341ed61765099 I0225 14:50:50.018353 18395 raft_consensus.cc:1988] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099 [term 0 FOLLOWER]: Advancing to term 1 I0225 14:50:50.018339 18516 raft_consensus.cc:1988] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391 [term 0 FOLLOWER]: Advancing to term 1 I0225 14:50:50.020423 18516 raft_consensus.cc:1939] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.554s I0225 14:50:50.020488 18395 raft_consensus.cc:1939] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.559s I0225 14:50:50.022032 18395 raft_consensus.cc:1649] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713690c2ff104bd6a70fbfa8303fbe61 in term 1. I0225 14:50:50.022066 18516 raft_consensus.cc:1649] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713690c2ff104bd6a70fbfa8303fbe61 in term 1. I0225 14:50:50.023109 18226 leader_election.cc:346] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [CANDIDATE]: Term 1 election: Vote granted by peer c30b588e10004dcdb01341ed61765099 I0225 14:50:50.023396 18224 leader_election.cc:346] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [CANDIDATE]: Term 1 election: Vote granted by peer 0d18db0d3e6e410a9310469fb6927391 I0225 14:50:50.023564 18224 leader_election.cc:247] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0225 14:50:50.024081 18616 raft_consensus.cc:1939] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.642s I0225 14:50:50.024310 18616 raft_consensus.cc:1831] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 1 FOLLOWER]: Leader election won for term 1 I0225 14:50:50.025372 18616 raft_consensus.cc:442] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 1 LEADER]: Becoming Leader. State: Replica: 713690c2ff104bd6a70fbfa8303fbe61, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0225 14:50:50.026321 18616 consensus_queue.cc:145] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" member_type: VOTER last_known_addr { host: "127.109.54.0" port: 33864 } } peers { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" member_type: VOTER last_known_addr { host: "127.109.54.2" port: 59678 } } peers { permanent_uuid: "c30b588e10004dcdb01341ed61765099" member_type: VOTER last_known_addr { host: "127.109.54.1" port: 44997 } } I0225 14:50:50.043742 18083 catalog_manager.cc:1554] Tablet: 5057465b8ee44d26ad2df63c369fbff3 reported consensus state change. New consensus state: current_term: 1 leader_uuid: "713690c2ff104bd6a70fbfa8303fbe61" config { opid_index: -1 local: false peers { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" member_type: VOTER last_known_addr { host: "127.109.54.0" port: 33864 } } peers { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" member_type: VOTER last_known_addr { host: "127.109.54.2" port: 59678 } } peers { permanent_uuid: "c30b588e10004dcdb01341ed61765099" member_type: VOTER last_known_addr { host: "127.109.54.1" port: 44997 } } } W0225 14:50:50.044764 18625 raft_consensus_state.cc:524] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0225 14:50:50.081324 27958 master_failover-itest.cc:250] Pausing leader master I0225 14:50:50.532618 18517 raft_consensus.cc:831] T 5057465b8ee44d26ad2df63c369fbff3 P 0d18db0d3e6e410a9310469fb6927391 [term 1 FOLLOWER]: Refusing update from remote peer 713690c2ff104bd6a70fbfa8303fbe61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0225 14:50:50.533186 18396 raft_consensus.cc:831] T 5057465b8ee44d26ad2df63c369fbff3 P c30b588e10004dcdb01341ed61765099 [term 1 FOLLOWER]: Refusing update from remote peer 713690c2ff104bd6a70fbfa8303fbe61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0225 14:50:50.534813 18626 consensus_queue.cc:572] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [LEADER]: Connected to new peer: Peer: 0d18db0d3e6e410a9310469fb6927391, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 14:50:50.535089 18616 consensus_queue.cc:572] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [LEADER]: Connected to new peer: Peer: c30b588e10004dcdb01341ed61765099, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0225 14:50:50.554569 18629 raft_consensus_state.cc:517] T 5057465b8ee44d26ad2df63c369fbff3 P 713690c2ff104bd6a70fbfa8303fbe61 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0225 14:50:50.562244 18219 raft_consensus.cc:358] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 1 FOLLOWER]: Failure of leader 3e305ed53fac466ba564ca1e7586e84a detected. Triggering leader election I0225 14:50:50.562414 18219 raft_consensus.cc:1988] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 1 FOLLOWER]: Advancing to term 2 I0225 14:50:50.564147 18219 raft_consensus.cc:1939] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.513s I0225 14:50:50.564288 18219 raft_consensus.cc:378] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:50:50.567947 18219 leader_election.cc:222] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 2 election: Requesting vote from peer 3e305ed53fac466ba564ca1e7586e84a I0225 14:50:50.568414 18219 leader_election.cc:222] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 2 election: Requesting vote from peer f9bc2ec9795e45f99414d3460b8af958 I0225 14:50:50.569885 18185 raft_consensus.cc:1988] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 1 FOLLOWER]: Advancing to term 2 I0225 14:50:50.571604 18185 raft_consensus.cc:1939] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.503s I0225 14:50:50.573035 18185 raft_consensus.cc:1649] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af53ee3fc638416993de9457afc302c5 in term 2. I0225 14:50:50.573817 18121 leader_election.cc:346] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 2 election: Vote granted by peer f9bc2ec9795e45f99414d3460b8af958 I0225 14:50:50.574025 18121 leader_election.cc:247] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0225 14:50:50.575026 18630 raft_consensus.cc:1939] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.531s I0225 14:50:50.575302 18630 raft_consensus.cc:1831] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 FOLLOWER]: Leader election won for term 2 I0225 14:50:50.576081 18630 raft_consensus.cc:442] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 LEADER]: Becoming Leader. State: Replica: af53ee3fc638416993de9457afc302c5, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 8 Committed: term: 1 index: 8} I0225 14:50:50.576967 18630 consensus_queue.cc:145] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 1.8, Majority replicated op: 1.8, Committed index: 1.8, Last appended: 1.8, Current term: 2, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:50:50.581223 18631 sys_catalog.cc:227] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af53ee3fc638416993de9457afc302c5. Latest consensus state: current_term: 2 leader_uuid: "af53ee3fc638416993de9457afc302c5" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:50.581599 18631 sys_catalog.cc:230] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER W0225 14:50:50.582800 18634 raft_consensus_state.cc:524] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 1 index: 8, New majority replicated is: term: 1 index: 8, Current term is: 2 I0225 14:50:50.583351 18635 catalog_manager.cc:520] Loading table and tablet metadata into memory... I0225 14:50:50.590898 18635 catalog_manager.cc:202] Loaded metadata for table [id=07c5ef70986145e9ab072daea8b0cf20] I0225 14:50:50.594540 18635 catalog_manager.cc:266] Loaded metadata for tablet 5057465b8ee44d26ad2df63c369fbff3 (table testAlterTableSync [id=07c5ef70986145e9ab072daea8b0cf20]) W0225 14:50:51.082420 18121 leader_election.cc:274] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e305ed53fac466ba564ca1e7586e84a: Timed out: RequestConsensusVote RPC to 127.0.0.1:11010 timed out after 0.513s I0225 14:50:51.082451 18186 raft_consensus.cc:831] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 2 FOLLOWER]: Refusing update from remote peer af53ee3fc638416993de9457afc302c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I0225 14:50:51.084071 18630 consensus_queue.cc:572] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [LEADER]: Connected to new peer: Peer: f9bc2ec9795e45f99414d3460b8af958, Is new: false, Last received: 1.8, Next index: 9, Last known committed idx: 7, Last exchange result: ERROR, Needs remote bootstrap: false I0225 14:50:51.088793 18637 sys_catalog.cc:227] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af53ee3fc638416993de9457afc302c5. Latest consensus state: current_term: 2 leader_uuid: "af53ee3fc638416993de9457afc302c5" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:51.089311 18637 sys_catalog.cc:230] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:50:51.091425 18639 raft_consensus_state.cc:517] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 1 index: 8 New committed index is: term: 2 index: 9 I0225 14:50:51.094362 18637 sys_catalog.cc:227] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "af53ee3fc638416993de9457afc302c5" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:50:51.094677 18637 sys_catalog.cc:230] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER W0225 14:50:52.081794 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 1 times. W0225 14:50:53.385056 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 2 times. W0225 14:50:54.805938 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 3 times. W0225 14:50:55.083163 27958 client-internal.cc:178] Unable to send the request (table { table_name: "testAlterTableSync" } new_table_name: "testAlterTableSyncRenamed") to leader Master (127.0.0.1:11010): Timed out: AlterTable RPC to 127.0.0.1:11010 timed out after 5.000s I0225 14:50:55.083379 27958 client-internal.cc:182] Determining the new leader Master and retrying... I0225 14:50:55.089372 18129 catalog_manager.cc:1083] Servicing AlterTable request from {real_user=adar, eff_user=} at 127.0.0.1:35640: table { table_name: "testAlterTableSync" } new_table_name: "testAlterTableSyncRenamed" W0225 14:50:55.108788 18129 catalog_manager.cc:1801] Unable to reset TS proxy: Not found: no leader W0225 14:50:55.109128 18129 catalog_manager.cc:2417] Failed to send alter table request: Not found: Failed to reset TS proxy: no leader W0225 14:50:56.306329 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 4 times. W0225 14:50:57.614930 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 5 times. W0225 14:50:59.116283 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 6 times. W0225 14:51:00.083171 18457 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11010: Timed out: Failed to send heartbeat: TSHeartbeat RPC to 127.0.0.1:11010 timed out after 10.000s I0225 14:51:00.088479 18457 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 I0225 14:51:00.090798 18133 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "c30b588e10004dcdb01341ed61765099" instance_seqno: 1456440648977430 } as {real_user=adar, eff_user=} at 127.109.54.1:50279; Asking this server to re-register. I0225 14:51:00.091871 18457 heartbeater.cc:328] Registering TS with master... I0225 14:51:00.092036 18457 heartbeater.cc:334] Sending a full tablet report to master... W0225 14:51:00.092464 18578 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11010: Timed out: Failed to send heartbeat: TSHeartbeat RPC to 127.0.0.1:11010 timed out after 10.000s I0225 14:51:00.094135 18134 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "c30b588e10004dcdb01341ed61765099" instance_seqno: 1456440648977430 } with Master I0225 14:51:00.095101 18134 catalog_manager.cc:1453] TS c30b588e10004dcdb01341ed61765099 does not have the latest schema for tablet 5057465b8ee44d26ad2df63c369fbff3 (table testAlterTableSyncRenamed [id=07c5ef70986145e9ab072daea8b0cf20]). Expected version 1 got 0 I0225 14:51:00.097049 18578 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 I0225 14:51:00.098794 18127 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" instance_seqno: 1456440649401216 } as {real_user=adar, eff_user=} at 127.109.54.2:46583; Asking this server to re-register. I0225 14:51:00.099711 18578 heartbeater.cc:328] Registering TS with master... I0225 14:51:00.099912 18578 heartbeater.cc:334] Sending a full tablet report to master... I0225 14:51:00.101845 18128 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "0d18db0d3e6e410a9310469fb6927391" instance_seqno: 1456440649401216 } with Master W0225 14:51:00.108217 18134 catalog_manager.cc:1801] Unable to reset TS proxy: Not found: no leader W0225 14:51:00.108340 18134 catalog_manager.cc:2417] Failed to send alter table request: Not found: Failed to reset TS proxy: no leader I0225 14:51:00.108381 18128 catalog_manager.cc:1453] TS 0d18db0d3e6e410a9310469fb6927391 does not have the latest schema for tablet 5057465b8ee44d26ad2df63c369fbff3 (table testAlterTableSyncRenamed [id=07c5ef70986145e9ab072daea8b0cf20]). Expected version 1 got 0 W0225 14:51:00.117671 18128 catalog_manager.cc:1801] Unable to reset TS proxy: Not found: no leader W0225 14:51:00.117789 18128 catalog_manager.cc:2417] Failed to send alter table request: Not found: Failed to reset TS proxy: no leader W0225 14:51:00.558116 18333 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11010: Timed out: Failed to send heartbeat: TSHeartbeat RPC to 127.0.0.1:11010 timed out after 10.000s W0225 14:51:00.617267 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 7 times. I0225 14:51:00.875164 18214 raft_consensus.cc:358] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 2 FOLLOWER]: Failure of leader af53ee3fc638416993de9457afc302c5 detected. Triggering leader election I0225 14:51:00.875401 18214 raft_consensus.cc:1988] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 2 FOLLOWER]: Advancing to term 3 I0225 14:51:00.877303 18214 raft_consensus.cc:1939] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.514s I0225 14:51:00.877419 18214 raft_consensus.cc:378] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 3 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:51:00.881008 18214 leader_election.cc:222] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [CANDIDATE]: Term 3 election: Requesting vote from peer 3e305ed53fac466ba564ca1e7586e84a I0225 14:51:00.881446 18214 leader_election.cc:222] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [CANDIDATE]: Term 3 election: Requesting vote from peer af53ee3fc638416993de9457afc302c5 I0225 14:51:00.883083 18137 raft_consensus.cc:1612] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 LEADER]: Leader election vote request: Denying vote to candidate f9bc2ec9795e45f99414d3460b8af958 for term 3 because replica is either leader or believes a valid leader to be alive. I0225 14:51:00.884174 18166 leader_election.cc:360] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [CANDIDATE]: Term 3 election: Vote denied by peer af53ee3fc638416993de9457afc302c5. Message: Invalid argument: T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 LEADER]: Leader election vote request: Denying vote to candidate f9bc2ec9795e45f99414d3460b8af958 for term 3 because replica is either leader or believes a valid leader to be alive. I0225 14:51:01.063299 18333 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 I0225 14:51:01.064728 18125 master_service.cc:172] Got heartbeat from unknown tablet server { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" instance_seqno: 1456440648596239 } as {real_user=adar, eff_user=} at 127.109.54.0:34007; Asking this server to re-register. I0225 14:51:01.065603 18333 heartbeater.cc:328] Registering TS with master... I0225 14:51:01.065831 18333 heartbeater.cc:334] Sending a full tablet report to master... I0225 14:51:01.067625 18126 ts_manager.cc:82] Registered new tablet server { permanent_uuid: "713690c2ff104bd6a70fbfa8303fbe61" instance_seqno: 1456440648596239 } with Master I0225 14:51:01.068325 18126 catalog_manager.cc:1453] TS 713690c2ff104bd6a70fbfa8303fbe61 does not have the latest schema for tablet 5057465b8ee44d26ad2df63c369fbff3 (table testAlterTableSyncRenamed [id=07c5ef70986145e9ab072daea8b0cf20]). Expected version 1 got 0 I0225 14:51:01.073870 18185 raft_consensus.cc:799] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 3 FOLLOWER]: Rejecting Update request from peer af53ee3fc638416993de9457afc302c5 for earlier term 2. Current term is 3. Ops: [] I0225 14:51:01.075086 18631 consensus_queue.cc:582] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [LEADER]: Peer responded invalid term: Peer: f9bc2ec9795e45f99414d3460b8af958, Is new: false, Last received: 2.12, Next index: 13, Last known committed idx: 12, Last exchange result: ERROR, Needs remote bootstrap: false I0225 14:51:01.075562 18654 raft_consensus.cc:1983] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 LEADER]: Stepping down as leader of term 2 I0225 14:51:01.075758 18654 raft_consensus.cc:477] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af53ee3fc638416993de9457afc302c5, State: 1, Role: LEADER Watermarks: {Received: term: 2 index: 13 Committed: term: 2 index: 12} I0225 14:51:01.076253 18654 consensus_queue.cc:162] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 2.12, Committed index: 2.12, Last appended: 2.13, Current term: 2, Majority size: -1, State: 1, Mode: NON_LEADER I0225 14:51:01.076927 18654 consensus_peers.cc:342] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer f9bc2ec9795e45f99414d3460b8af958 (127.0.0.1:11012): Closing peer: f9bc2ec9795e45f99414d3460b8af958 I0225 14:51:01.077549 18654 consensus_peers.cc:342] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Closing peer: 3e305ed53fac466ba564ca1e7586e84a I0225 14:51:01.078145 18654 raft_consensus.cc:1988] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 2 FOLLOWER]: Advancing to term 3 W0225 14:51:01.113323 18127 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Illegal state: Not the leader. Local UUID: af53ee3fc638416993de9457afc302c5, Consensus state: current_term: 3 config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } W0225 14:51:01.114078 18457 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11011: Service unavailable: master is no longer the leader W0225 14:51:01.122262 18134 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Illegal state: Not the leader. Local UUID: af53ee3fc638416993de9457afc302c5, Consensus state: current_term: 3 config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } W0225 14:51:01.123056 18578 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11011: Service unavailable: master is no longer the leader W0225 14:51:01.396209 18166 leader_election.cc:274] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3e305ed53fac466ba564ca1e7586e84a: Timed out: RequestConsensusVote RPC to 127.0.0.1:11010 timed out after 0.514s I0225 14:51:01.396360 18166 leader_election.cc:247] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I0225 14:51:01.397374 18656 raft_consensus.cc:1939] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.524s I0225 14:51:01.397573 18656 raft_consensus.cc:1793] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given I0225 14:51:01.487134 27958 client-internal.cc:198] Determining the new leader Master and retrying... I0225 14:51:01.591372 18219 raft_consensus.cc:362] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 3 FOLLOWER]: No leader contacted us within the election timeout. Triggering leader election I0225 14:51:01.591487 18219 raft_consensus.cc:1988] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 3 FOLLOWER]: Advancing to term 4 I0225 14:51:01.593013 18219 raft_consensus.cc:1939] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.513s I0225 14:51:01.593158 18219 raft_consensus.cc:378] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 4 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:51:01.595232 18219 leader_election.cc:222] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 4 election: Requesting vote from peer 3e305ed53fac466ba564ca1e7586e84a I0225 14:51:01.595574 18219 leader_election.cc:222] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 4 election: Requesting vote from peer f9bc2ec9795e45f99414d3460b8af958 I0225 14:51:01.596657 18186 raft_consensus.cc:1988] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 3 FOLLOWER]: Advancing to term 4 I0225 14:51:01.599086 18186 raft_consensus.cc:1939] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.546s I0225 14:51:01.600491 18186 raft_consensus.cc:1649] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af53ee3fc638416993de9457afc302c5 in term 4. I0225 14:51:01.601126 18121 leader_election.cc:346] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 4 election: Vote granted by peer f9bc2ec9795e45f99414d3460b8af958 I0225 14:51:01.601274 18121 leader_election.cc:247] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I0225 14:51:01.602072 18657 raft_consensus.cc:1939] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 4 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 0.575s I0225 14:51:01.602264 18657 raft_consensus.cc:1831] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 4 FOLLOWER]: Leader election won for term 4 I0225 14:51:01.602999 18657 raft_consensus.cc:442] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 4 LEADER]: Becoming Leader. State: Replica: af53ee3fc638416993de9457afc302c5, State: 1, Role: LEADER Watermarks: {Received: term: 2 index: 13 Committed: term: 2 index: 12} I0225 14:51:01.603210 18657 consensus_queue.cc:145] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [LEADER]: Queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 2.12, Committed index: 2.12, Last appended: 2.13, Current term: 4, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } I0225 14:51:01.605270 18658 sys_catalog.cc:227] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af53ee3fc638416993de9457afc302c5. Latest consensus state: current_term: 4 leader_uuid: "af53ee3fc638416993de9457afc302c5" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:51:01.605648 18658 sys_catalog.cc:230] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER W0225 14:51:01.606482 18661 raft_consensus_state.cc:524] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 4 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 12, New majority replicated is: term: 2 index: 12, Current term is: 4 I0225 14:51:01.606959 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.607028 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.608664 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.608705 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.609560 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.609598 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.610558 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.610596 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.611665 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.611701 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.612855 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.612895 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.614336 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.614384 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.616065 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.616122 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.618105 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.618155 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.618805 18457 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 W0225 14:51:01.620008 18132 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Service unavailable: Leader not yet ready to serve requests I0225 14:51:01.620522 18662 transaction_tracker.cc:230] Dumping currently running transactions: W0225 14:51:01.620725 18457 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11011: Service unavailable: master is no longer the leader I0225 14:51:01.620592 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.623450 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.623520 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.626948 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.627019 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.627769 18578 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 W0225 14:51:01.629075 18127 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Service unavailable: Leader not yet ready to serve requests W0225 14:51:01.629787 18578 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11011: Service unavailable: master is no longer the leader I0225 14:51:01.631150 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.631198 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.636286 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.636337 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.642523 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.642571 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.650230 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.650279 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.659667 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.659716 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.671308 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.671363 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.685711 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.685783 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.703584 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.703634 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.725778 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.725827 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.753330 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.753381 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.787701 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.787772 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.830430 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.830478 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.883673 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.883723 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:01.950100 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:01.950150 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:02.032922 18662 transaction_tracker.cc:230] Dumping currently running transactions: I0225 14:51:02.032971 18662 transaction_tracker.cc:232] R-P WriteTransaction [type=LEADER, start_time=2016-02-25 14:51:01, state=WriteTransactionState 0x6120002ffd40 [op_id=(), ts=5965580947747491840, rows=[MUTATE (int8 entry_type=2, string entry_id=5057465b8ee44d26ad2df63c369fbff3) SET metadata=\032â\001\010\001\022 713690c2ff104bd6a70fbfa8303fbe61\032»\001\010ÿÿÿÿÿÿÿÿÿ\001\020\000\0328\n 713690c2ff104bd6a70fbfa8303fbe61\020\001\032\022\n\014127.109.54.0\020Ȉ\002\0328\n 0d18db0d3e6e410a9310469fb6927391\020\001\032\022\n\014127.109.54.2\020žÒ\003\0328\n c30b588e10004dcdb01341ed61765099\020\001\032\022\n\014127.109.54.1\020Åß\002 \002*%Tablet reported with an active leader2 07c5ef70986145e9ab072daea8b0cf20:\004\022\000\032\000]]] I0225 14:51:02.106441 18187 raft_consensus.cc:831] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 4 FOLLOWER]: Refusing update from remote peer af53ee3fc638416993de9457afc302c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 4 index: 14. (index mismatch) I0225 14:51:02.107586 18657 consensus_queue.cc:572] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [LEADER]: Connected to new peer: Peer: f9bc2ec9795e45f99414d3460b8af958, Is new: false, Last received: 2.12, Next index: 13, Last known committed idx: 12, Last exchange result: ERROR, Needs remote bootstrap: false W0225 14:51:02.109742 18121 leader_election.cc:274] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 3e305ed53fac466ba564ca1e7586e84a: Timed out: RequestConsensusVote RPC to 127.0.0.1:11010 timed out after 0.513s I0225 14:51:02.112884 18663 sys_catalog.cc:227] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af53ee3fc638416993de9457afc302c5. Latest consensus state: current_term: 4 leader_uuid: "af53ee3fc638416993de9457afc302c5" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:51:02.113309 18663 sys_catalog.cc:230] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:51:02.115866 18665 raft_consensus_state.cc:517] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 4 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 2 index: 12 New committed index is: term: 4 index: 14 I0225 14:51:02.119154 18663 sys_catalog.cc:227] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "af53ee3fc638416993de9457afc302c5" config { opid_index: -1 local: false peers { permanent_uuid: "3e305ed53fac466ba564ca1e7586e84a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11010 } } peers { permanent_uuid: "af53ee3fc638416993de9457afc302c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11011 } } peers { permanent_uuid: "f9bc2ec9795e45f99414d3460b8af958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 11012 } } } I0225 14:51:02.119454 18663 sys_catalog.cc:230] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [sys.catalog]: This master's current role is: FOLLOWER, previous role was: FOLLOWER I0225 14:51:02.125360 18457 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 W0225 14:51:02.126647 18129 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Service unavailable: Leader not yet ready to serve requests W0225 14:51:02.127418 18457 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11011: Service unavailable: master is no longer the leader I0225 14:51:02.134156 18578 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 W0225 14:51:02.135416 18132 master_service.cc:142] Received a heartbeat, but this Master instance is not a leader or a single Master: Service unavailable: Leader not yet ready to serve requests W0225 14:51:02.136098 18578 heartbeater.cc:411] Failed to heartbeat to 127.0.0.1:11011: Service unavailable: master is no longer the leader I0225 14:51:02.136443 18662 catalog_manager.cc:520] Loading table and tablet metadata into memory... I0225 14:51:02.138829 18673 tablet.cc:706] Alter schema from Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NOT NULL] ] version 0 to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NOT NULL] ] version 1 I0225 14:51:02.139905 18662 catalog_manager.cc:202] Loaded metadata for table [id=07c5ef70986145e9ab072daea8b0cf20] I0225 14:51:02.141629 18674 tablet.cc:706] Alter schema from Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NOT NULL] ] version 0 to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NOT NULL] ] version 1 I0225 14:51:02.143271 18662 catalog_manager.cc:266] Loaded metadata for tablet 5057465b8ee44d26ad2df63c369fbff3 (table testAlterTableSyncRenamed [id=07c5ef70986145e9ab072daea8b0cf20]) I0225 14:51:02.157290 18662 catalog_manager.cc:2625] testAlterTableSyncRenamed [id=07c5ef70986145e9ab072daea8b0cf20] - Alter table completed version=1 I0225 14:51:02.390205 18676 tablet.cc:706] Alter schema from Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NOT NULL] ] version 0 to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NOT NULL] ] version 1 I0225 14:51:02.632027 18457 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 I0225 14:51:02.641144 18578 heartbeater.cc:258] Connected to a leader master server at 127.0.0.1:11011 W0225 14:51:03.105592 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 1 times. W0225 14:51:04.605576 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 2 times. W0225 14:51:05.876183 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 3 times. W0225 14:51:06.490028 27958 client-internal.cc:200] Unable to determine the new leader Master: Timed out: GetMasterRegistration RPC to 127.0.0.1:11010 timed out after 5.000s W0225 14:51:07.377065 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 4 times. W0225 14:51:08.876981 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 5 times. W0225 14:51:10.378217 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 6 times. W0225 14:51:11.879072 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 7 times. W0225 14:51:13.379204 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 8 times. W0225 14:51:14.881160 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 9 times. W0225 14:51:16.381260 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 10 times. W0225 14:51:17.881868 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 11 times. W0225 14:51:19.383247 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 12 times. W0225 14:51:20.883214 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 13 times. W0225 14:51:22.383857 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 14 times. W0225 14:51:23.885177 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 15 times. W0225 14:51:25.386258 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 16 times. W0225 14:51:26.887388 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 17 times. W0225 14:51:28.387086 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 18 times. W0225 14:51:29.888054 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 19 times. W0225 14:51:31.389540 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 20 times. W0225 14:51:32.889417 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 21 times. W0225 14:51:34.390774 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 22 times. W0225 14:51:35.891616 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 23 times. W0225 14:51:37.392783 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 24 times. W0225 14:51:38.892673 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 25 times. W0225 14:51:40.394031 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 26 times. W0225 14:51:41.894393 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 27 times. W0225 14:51:43.395503 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 28 times. W0225 14:51:44.896544 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 29 times. W0225 14:51:46.397457 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 30 times. W0225 14:51:47.898550 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 31 times. W0225 14:51:49.398758 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 32 times. W0225 14:51:50.899551 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 33 times. W0225 14:51:52.400166 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 34 times. W0225 14:51:53.900810 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 35 times. W0225 14:51:55.402035 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 36 times. W0225 14:51:56.902781 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 37 times. W0225 14:51:58.403228 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 38 times. W0225 14:51:59.904721 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 39 times. W0225 14:52:01.404907 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 40 times. W0225 14:52:02.905753 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 41 times. W0225 14:52:04.406513 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 42 times. W0225 14:52:05.906877 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 43 times. W0225 14:52:07.408133 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 44 times. W0225 14:52:08.909057 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 45 times. W0225 14:52:10.409734 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 46 times. W0225 14:52:11.910290 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 47 times. W0225 14:52:13.411346 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 48 times. W0225 14:52:14.912137 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 49 times. W0225 14:52:16.412856 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 50 times. W0225 14:52:17.913055 18121 consensus_peers.cc:319] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 -> Peer 3e305ed53fac466ba564ca1e7586e84a (127.0.0.1:11010): Couldn't send request to peer 3e305ed53fac466ba564ca1e7586e84a for tablet 00000000000000000000000000000000 Status: Timed out: UpdateConsensus RPC to 127.0.0.1:11010 timed out after 1.000s. Retrying in the next heartbeat period. Already tried 51 times. /home/adar/kudu/src/kudu/integration-tests/master_failover-itest.cc:255: Failure Failed Bad status: Timed out: Timed out waiting for AlterTable terminate called after throwing an instance of 'testing::internal::GoogleTestFailureException' W0225 14:52:18.131186 18471 connection.cc:400] client connection to 127.0.0.1:11010 recv error: Network error: recv error: Connection reset by peer (error 104) W0225 14:52:18.131189 18350 connection.cc:400] client connection to 127.0.0.1:11010 recv error: Network error: recv error: Connection reset by peer (error 104) what(): /home/adar/kudu/src/kudu/integration-tests/master_failover-itest.cc:255: Failure Failed Bad status: Timed out: Timed out waiting for AlterTable *** Aborted at 1456440738 (unix time) try "date -d @1456440738" if you are using GNU date *** W0225 14:52:18.131860 18121 connection.cc:400] client connection to 127.0.0.1:11010 recv error: Network error: recv error: Connection reset by peer (error 104) W0225 14:52:18.132191 18166 connection.cc:400] client connection to 127.0.0.1:11010 recv error: Network error: recv error: Connection reset by peer (error 104) PC: @ 0x3ae0a32625 (unknown) *** SIGABRT (@0x10ba00006d36) received by PID 27958 (TID 0x7fede1f0a8e0) from PID 27958; stack trace: *** W0225 14:52:18.132400 18226 connection.cc:400] client connection to 127.0.0.1:11010 recv error: Network error: recv error: Connection reset by peer (error 104) @ 0x3ae0e0f710 (unknown) at ??:0 W0225 14:52:18.132712 18589 connection.cc:400] client connection to 127.0.0.1:11010 recv error: Network error: recv error: Connection reset by peer (error 104) @ 0x3ae0a32625 (unknown) at ??:0 @ 0x3ae0a33e05 (unknown) at ??:0 @ 0x3ae3abea7d (unknown) at ??:0 @ 0x3ae3abcbd6 (unknown) at ??:0 @ 0x3ae3abcc03 (unknown) at ??:0 @ 0x3ae3abcc86 (unknown) at ??:0 @ 0x7fede4fff172 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0 @ 0x7fede4ff37dd testing::UnitTest::Run() at ??:0 @ 0x7fedf2f5fb90 RUN_ALL_TESTS() at ??:0 I0225 14:52:18.137388 18138 raft_consensus.cc:799] T 00000000000000000000000000000000 P af53ee3fc638416993de9457afc302c5 [term 4 LEADER]: Rejecting Update request from peer 3e305ed53fac466ba564ca1e7586e84a for earlier term 1. Current term is 4. Ops: [] @ 0x7fedf2f5dd9b main at ??:0 I0225 14:52:18.138530 18184 raft_consensus.cc:799] T 00000000000000000000000000000000 P f9bc2ec9795e45f99414d3460b8af958 [term 4 FOLLOWER]: Rejecting Update request from peer 3e305ed53fac466ba564ca1e7586e84a for earlier term 1. Current term is 4. Ops: [] @ 0x3ae0a1ed5d (unknown) at ??:0 I0225 14:52:18.138558 18726 consensus_queue.cc:582] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [LEADER]: Peer responded invalid term: Peer: af53ee3fc638416993de9457afc302c5, Is new: false, Last received: 4.15, Next index: 16, Last known committed idx: 15, Last exchange result: ERROR, Needs remote bootstrap: false @ 0x428df5 (unknown) at ??:0 I0225 14:52:18.139508 18735 raft_consensus.cc:1983] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 LEADER]: Stepping down as leader of term 1 I0225 14:52:18.139519 18726 consensus_queue.cc:582] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [LEADER]: Peer responded invalid term: Peer: f9bc2ec9795e45f99414d3460b8af958, Is new: false, Last received: 4.15, Next index: 16, Last known committed idx: 15, Last exchange result: ERROR, Needs remote bootstrap: false I0225 14:52:18.139619 18735 raft_consensus.cc:477] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e305ed53fac466ba564ca1e7586e84a, State: 1, Role: LEADER Watermarks: {Received: term: 1 index: 8 Committed: term: 1 index: 8} I0225 14:52:18.140115 18735 consensus_queue.cc:162] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated op: 1.8, Majority replicated op: 1.8, Committed index: 1.8, Last appended: 1.8, Current term: 1, Majority size: -1, State: 1, Mode: NON_LEADER I0225 14:52:18.140743 18735 consensus_peers.cc:342] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a -> Peer f9bc2ec9795e45f99414d3460b8af958 (127.0.0.1:11012): Closing peer: f9bc2ec9795e45f99414d3460b8af958 I0225 14:52:18.141487 18735 consensus_peers.cc:342] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a -> Peer af53ee3fc638416993de9457afc302c5 (127.0.0.1:11011): Closing peer: af53ee3fc638416993de9457afc302c5 I0225 14:52:18.142200 18735 raft_consensus.cc:1988] T 00000000000000000000000000000000 P 3e305ed53fac466ba564ca1e7586e84a [term 1 FOLLOWER]: Advancing to term 4 *** Aborted at 1456440738 (unix time) try "date -d @1456440738" if you are using GNU date *** *** Aborted at 1456440738 (unix time) try "date -d @1456440738" if you are using GNU date *** *** Aborted at 1456440738 (unix time) try "date -d @1456440738" if you are using GNU date *** *** Aborted at 1456440738 (unix time) try "date -d @1456440738" if you are using GNU date *** *** Aborted at 1456440738 (unix time) try "date -d @1456440738" if you are using GNU date *** *** Aborted at 1456440738 (unix time) try "date -d @1456440738" if you are using GNU date *** PC: @ 0x3ae0e0ef3d (unknown) PC: @ 0x3ae0e0ef3d (unknown) *** SIGTERM (@0x10ba00006d36) received by PID 18113 (TID 0x7f1e74ad1820) from PID 27958; stack trace: *** *** SIGTERM (@0x10ba00006d36) received by PID 18158 (TID 0x7f11e8cb8820) from PID 27958; stack trace: *** PC: @ 0x3ae0e0ef3d (unknown) @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 *** SIGTERM (@0x10ba00006d36) received by PID 18339 (TID 0x7f0a78e35820) from PID 27958; stack trace: *** PC: @ 0x3ae0e0ef3d (unknown) PC: @ 0x3ae0e0ef3d (unknown) PC: @ 0x3ae0e0ef3d (unknown) @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 *** SIGTERM (@0x10ba00006d36) received by PID 18463 (TID 0x7f1751663820) from PID 27958; stack trace: *** *** SIGTERM (@0x10ba00006d36) received by PID 18204 (TID 0x7fe9f3490820) from PID 27958; stack trace: *** *** SIGTERM (@0x10ba00006d36) received by PID 18071 (TID 0x7f6df639d820) from PID 27958; stack trace: *** @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0f710 (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x7f11eae5a1c8 base::SleepForNanoseconds() at ??:0 @ 0x7f1e76c731c8 base::SleepForNanoseconds() at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x3ae0e0ef3d (unknown) at ??:0 @ 0x7f0a7a7001c8 base::SleepForNanoseconds() at ??:0 @ 0x7f1752f2e1c8 base::SleepForNanoseconds() at ??:0 @ 0x7f11ec1737be kudu::SleepFor() at ??:0 @ 0x7f1e77f8c7be kudu::SleepFor() at ??:0 @ 0x4ea327 kudu::master::MasterMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:287 @ 0x4ea327 kudu::master::MasterMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:287 @ 0x4e7f3e main at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:311 @ 0x4e7f3e main at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:311 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x41dd45 (unknown) at ??:0 @ 0x41dd45 (unknown) at ??:0 @ 0x7f6df853f1c8 base::SleepForNanoseconds() at ??:0 @ 0x7fe9f4d5b1c8 base::SleepForNanoseconds() at ??:0 @ 0x7f0a7ba187be kudu::SleepFor() at ??:0 @ 0x4ea36e kudu::tserver::TabletServerMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:503 @ 0x7f17542467be kudu::SleepFor() at ??:0 @ 0x4ea36e kudu::tserver::TabletServerMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:503 @ 0x4e7f5e main at crtstuff.c:0 @ 0x4e7f5e main at crtstuff.c:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x41dd65 (unknown) at ??:0 @ 0x41dd65 (unknown) at ??:0 @ 0x7f6df98587be kudu::SleepFor() at ??:0 @ 0x4ea327 kudu::master::MasterMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:287 @ 0x7fe9f60737be kudu::SleepFor() at ??:0 @ 0x4ea36e kudu::tserver::TabletServerMain() at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/lsan/lsan_common.cc:503 @ 0x4e7f3e main at /home/adar/kudu/thirdparty/llvm-3.7.1.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:311 @ 0x4e7f5e main at crtstuff.c:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x3ae0a1ed5d (unknown) at ??:0 @ 0x41dd45 (unknown) at ??:0 @ 0x41dd65 (unknown) at ??:0