Running raft_consensus-itest, redirecting output into /home/todd/run_tha_testHSjhNC/build/test-logs/raft_consensus-itest.txt (attempt 1/1) ~/run_tha_testHSjhNC /home/todd/run_tha_testHSjhNC/build/fastdebug/raft_consensus-itest: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) /home/todd/run_tha_testHSjhNC/build/fastdebug/raft_consensus-itest: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) /home/todd/run_tha_testHSjhNC/build/fastdebug/raft_consensus-itest: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/thirdparty/installed/lib/libcurl.so.4) /home/todd/run_tha_testHSjhNC/build/fastdebug/raft_consensus-itest: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/thirdparty/installed/lib/libcurl.so.4) /home/todd/run_tha_testHSjhNC/build/fastdebug/raft_consensus-itest: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/thirdparty/installed/lib/libcurl.so.4) Note: This is test shard 1 of 8. [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from RaftConsensusITest [ RUN ] RaftConsensusITest.TestGetPermanentUuid I0529 14:48:37.620925 62863 test_util.cc:118] Using random seed: 1883689017 I0529 14:48:37.691246 62863 ts_itest-base.h:72] Starting cluster with: I0529 14:48:37.691551 62863 ts_itest-base.h:73] -------------- I0529 14:48:37.691778 62863 ts_itest-base.h:74] 3 tablet servers I0529 14:48:37.691995 62863 ts_itest-base.h:75] 3 replicas per TS I0529 14:48:37.692152 62863 ts_itest-base.h:76] -------------- I0529 14:48:37.697621 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-master kudu-master --master_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/master --master_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --master_web_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/master --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 kudu-master: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-master: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:37.891839 63007 master_main.cc:27] Initializing master server... I0529 14:48:37.907802 63007 server_base.cc:136] Could not load existing FS layout: Not found: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/master/instance: No such file or directory (error 2) I0529 14:48:37.908056 63007 server_base.cc:137] Creating new FS layout I0529 14:48:37.990804 63007 fs_manager.cc:270] Generated new instance metadata in path /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/master/instance: uuid: "7dd7efa5059e4b97ab64b7396d085c6c" format_stamp: "Formatted at 2015-05-29 21:48:37 on e1205.halxg.cloudera.com" I0529 14:48:38.080755 63007 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/master uuid: "7dd7efa5059e4b97ab64b7396d085c6c" format_stamp: "Formatted at 2015-05-29 21:48:37 on e1205.halxg.cloudera.com" I0529 14:48:38.161756 63007 master_main.cc:30] Starting Master server... I0529 14:48:38.161962 63007 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:38.220407 63007 rpc_server.cc:110] RPC server started. Bound to: 127.0.0.1:49200 I0529 14:48:38.222643 63007 webserver.cc:108] Starting webserver on localhost:0 I0529 14:48:38.222745 63007 webserver.cc:117] Document root disabled I0529 14:48:38.223965 63007 net_util.cc:149] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0529 14:48:38.225488 63007 webserver.cc:200] Webserver started. Bound to: http://127.0.0.1:49548/ I0529 14:48:38.226976 63007 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/master/info.pb I0529 14:48:38.234544 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-master as pid 63007 I0529 14:48:38.236711 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.245.143.0:0 --local_ip_for_outbound_sockets=127.245.143.0 --webserver_interface=127.245.143.0 --tablet_server_web_port=0 --tablet_server_master_addrs=127.0.0.1:49200 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:38.300336 63093 tablet_bootstrap.cc:311] Tablet 00000000000000000000000000000000: Bootstrap starting. I0529 14:48:38.305667 63093 tablet_bootstrap.cc:472] Time spent opening tablet 00000000000000000000000000000000: real 0.002s user 0.002s sys 0.000s I0529 14:48:38.306145 63093 tablet_bootstrap.cc:408] No blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I0529 14:48:38.308737 63093 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:38.314935 63093 tablet_bootstrap.cc:311] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I0529 14:48:38.319061 63093 local_consensus.cc:55] T 00000000000000000000000000000000 P 7dd7efa5059e4b97ab64b7396d085c6c: Starting LocalConsensus... I0529 14:48:38.321108 63093 sys_catalog.cc:195] T 00000000000000000000000000000000 P 7dd7efa5059e4b97ab64b7396d085c6c [sys.catalog]: SysCatalogTable state changed. New quorum config:opid_index: -1 local: true peers { permanent_uuid: "7dd7efa5059e4b97ab64b7396d085c6c" member_type: VOTER } I0529 14:48:38.321408 63093 sys_catalog.cc:198] T 00000000000000000000000000000000 P 7dd7efa5059e4b97ab64b7396d085c6c [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER I0529 14:48:38.325196 63093 sys_catalog.cc:263] T 00000000000000000000000000000000 P 7dd7efa5059e4b97ab64b7396d085c6c [sys.catalog]: configured and running, proceeding with master startup. I0529 14:48:38.325283 63118 catalog_manager.cc:451] Loading table and tablet metadata into memory... I0529 14:48:38.339860 63007 master_main.cc:33] Master server successfully started. I0529 14:48:38.423743 63094 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:38.440958 63094 server_base.cc:136] Could not load existing FS layout: Not found: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/instance: No such file or directory (error 2) I0529 14:48:38.441201 63094 server_base.cc:137] Creating new FS layout I0529 14:48:38.517617 63094 fs_manager.cc:270] Generated new instance metadata in path /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/instance: uuid: "c943855e209e48fb8496efe28c4d3859" format_stamp: "Formatted at 2015-05-29 21:48:38 on e1205.halxg.cloudera.com" I0529 14:48:38.587517 63094 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 uuid: "c943855e209e48fb8496efe28c4d3859" format_stamp: "Formatted at 2015-05-29 21:48:38 on e1205.halxg.cloudera.com" I0529 14:48:38.652292 63094 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:38.834269 63094 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.0:38750 I0529 14:48:38.836189 63094 webserver.cc:108] Starting webserver on 127.245.143.0:0 I0529 14:48:38.836308 63094 webserver.cc:117] Document root disabled I0529 14:48:38.838127 63094 webserver.cc:200] Webserver started. Bound to: http://127.245.143.0:56690/ I0529 14:48:38.839408 63094 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/info.pb I0529 14:48:38.840934 63094 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:38.841106 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 63094 I0529 14:48:38.842422 63094 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:38.842296 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.245.143.1:0 --local_ip_for_outbound_sockets=127.245.143.1 --webserver_interface=127.245.143.1 --tablet_server_web_port=0 --tablet_server_master_addrs=127.0.0.1:49200 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:39.031136 63315 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:39.047049 63315 server_base.cc:136] Could not load existing FS layout: Not found: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/instance: No such file or directory (error 2) I0529 14:48:39.047269 63315 server_base.cc:137] Creating new FS layout I0529 14:48:39.118880 63315 fs_manager.cc:270] Generated new instance metadata in path /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/instance: uuid: "547f9023097149b9ba85f8c1bbe37faf" format_stamp: "Formatted at 2015-05-29 21:48:39 on e1205.halxg.cloudera.com" I0529 14:48:39.189386 63315 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 uuid: "547f9023097149b9ba85f8c1bbe37faf" format_stamp: "Formatted at 2015-05-29 21:48:39 on e1205.halxg.cloudera.com" I0529 14:48:39.257426 63315 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:39.471279 63315 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.1:51298 I0529 14:48:39.474434 63315 webserver.cc:108] Starting webserver on 127.245.143.1:0 I0529 14:48:39.474611 63315 webserver.cc:117] Document root disabled I0529 14:48:39.477586 63315 webserver.cc:200] Webserver started. Bound to: http://127.245.143.1:54708/ I0529 14:48:39.479401 63315 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/info.pb I0529 14:48:39.481529 63315 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:39.483750 63315 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:39.488184 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 63315 I0529 14:48:39.488994 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.245.143.2:0 --local_ip_for_outbound_sockets=127.245.143.2 --webserver_interface=127.245.143.2 --tablet_server_web_port=0 --tablet_server_master_addrs=127.0.0.1:49200 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0529 14:48:39.489153 63311 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:49200 I0529 14:48:39.489878 63311 heartbeater.cc:312] Registering TS with master... I0529 14:48:39.490509 63311 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:39.494268 63062 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "c943855e209e48fb8496efe28c4d3859" instance_seqno: 1432936118795302 } with Master kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:39.682677 63472 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:39.700212 63472 server_base.cc:136] Could not load existing FS layout: Not found: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/instance: No such file or directory (error 2) I0529 14:48:39.700500 63472 server_base.cc:137] Creating new FS layout I0529 14:48:39.812347 63472 fs_manager.cc:270] Generated new instance metadata in path /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/instance: uuid: "c7db368a9e6f436cb7cde832e7f93875" format_stamp: "Formatted at 2015-05-29 21:48:39 on e1205.halxg.cloudera.com" I0529 14:48:39.891057 63472 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 uuid: "c7db368a9e6f436cb7cde832e7f93875" format_stamp: "Formatted at 2015-05-29 21:48:39 on e1205.halxg.cloudera.com" I0529 14:48:39.958684 63472 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:40.088742 63468 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:49200 I0529 14:48:40.089649 63468 heartbeater.cc:312] Registering TS with master... I0529 14:48:40.090337 63468 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:40.093387 63064 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "547f9023097149b9ba85f8c1bbe37faf" instance_seqno: 1432936119437569 } with Master I0529 14:48:40.182809 63472 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.2:45549 I0529 14:48:40.185328 63472 webserver.cc:108] Starting webserver on 127.245.143.2:0 I0529 14:48:40.185511 63472 webserver.cc:117] Document root disabled I0529 14:48:40.188623 63472 webserver.cc:200] Webserver started. Bound to: http://127.245.143.2:42528/ I0529 14:48:40.190505 63472 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.TestGetPermanentUuid.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/info.pb I0529 14:48:40.192773 63472 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:40.194969 63472 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:40.195145 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 63472 I0529 14:48:40.883589 63626 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:49200 I0529 14:48:40.884279 63626 heartbeater.cc:312] Registering TS with master... I0529 14:48:40.884894 63626 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:40.887506 63070 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "c7db368a9e6f436cb7cde832e7f93875" instance_seqno: 1432936120132261 } with Master I0529 14:48:40.891767 62863 external_mini_cluster.cc:285] 3 TS(s) registered with Master W0529 14:48:40.892396 63005 reactor.cc:200] Client call kudu.master.MasterService.ListTabletServers has no timeout set for connection id: {remote=127.0.0.1:49200, user_credentials={real_user=todd, eff_user=}} I0529 14:48:41.565853 63067 catalog_manager.cc:665] CreateTable from {real_user=todd, eff_user=} at 127.0.0.1:55645: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I0529 14:48:42.393400 63520 tablet_service.cc:420] Processing CreateTablet for tablet 7efc7c9a12544f9a94b4ddba475e889d (table=TestTable [id=a6afd6238ecd446da981378ed08b5666]), range=["", ""] I0529 14:48:42.394170 63376 tablet_service.cc:420] Processing CreateTablet for tablet 7efc7c9a12544f9a94b4ddba475e889d (table=TestTable [id=a6afd6238ecd446da981378ed08b5666]), range=["", ""] I0529 14:48:42.395369 63198 tablet_service.cc:420] Processing CreateTablet for tablet 7efc7c9a12544f9a94b4ddba475e889d (table=TestTable [id=a6afd6238ecd446da981378ed08b5666]), range=["", ""] I0529 14:48:42.479864 63198 ts_tablet_manager.cc:406] Registered tablet 7efc7c9a12544f9a94b4ddba475e889d I0529 14:48:42.482060 64003 ts_tablet_manager.cc:284] Bootstrapping tablet: 7efc7c9a12544f9a94b4ddba475e889d I0529 14:48:42.488236 64003 tablet_bootstrap.cc:311] Tablet 7efc7c9a12544f9a94b4ddba475e889d: Bootstrap starting. I0529 14:48:42.488456 63520 ts_tablet_manager.cc:406] Registered tablet 7efc7c9a12544f9a94b4ddba475e889d I0529 14:48:42.488979 63376 ts_tablet_manager.cc:406] Registered tablet 7efc7c9a12544f9a94b4ddba475e889d I0529 14:48:42.490973 64005 ts_tablet_manager.cc:284] Bootstrapping tablet: 7efc7c9a12544f9a94b4ddba475e889d I0529 14:48:42.491701 64006 ts_tablet_manager.cc:284] Bootstrapping tablet: 7efc7c9a12544f9a94b4ddba475e889d I0529 14:48:42.495110 64003 tablet_bootstrap.cc:472] Time spent opening tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.002s user 0.002s sys 0.000s I0529 14:48:42.496424 64003 tablet_bootstrap.cc:408] No blocks or log segments found for tablet: 7efc7c9a12544f9a94b4ddba475e889d creating new one. I0529 14:48:42.497208 64006 tablet_bootstrap.cc:311] Tablet 7efc7c9a12544f9a94b4ddba475e889d: Bootstrap starting. I0529 14:48:42.497402 64005 tablet_bootstrap.cc:311] Tablet 7efc7c9a12544f9a94b4ddba475e889d: Bootstrap starting. I0529 14:48:42.499167 64003 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:42.502508 64006 tablet_bootstrap.cc:472] Time spent opening tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.002s user 0.000s sys 0.001s I0529 14:48:42.503015 64006 tablet_bootstrap.cc:408] No blocks or log segments found for tablet: 7efc7c9a12544f9a94b4ddba475e889d creating new one. I0529 14:48:42.505496 64005 tablet_bootstrap.cc:472] Time spent opening tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.002s user 0.001s sys 0.001s I0529 14:48:42.505939 64006 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:42.506505 64003 tablet_bootstrap.cc:311] Tablet 7efc7c9a12544f9a94b4ddba475e889d: No bootstrap required, opened a new log I0529 14:48:42.506700 64005 tablet_bootstrap.cc:408] No blocks or log segments found for tablet: 7efc7c9a12544f9a94b4ddba475e889d creating new one. I0529 14:48:42.509528 64003 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.027s user 0.018s sys 0.008s I0529 14:48:42.510299 64005 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:42.513106 64006 tablet_bootstrap.cc:311] Tablet 7efc7c9a12544f9a94b4ddba475e889d: No bootstrap required, opened a new log I0529 14:48:42.516474 64006 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.024s user 0.015s sys 0.007s I0529 14:48:42.516665 64003 raft_consensus.cc:182] T 7efc7c9a12544f9a94b4ddba475e889d P c943855e209e48fb8496efe28c4d3859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "c7db368a9e6f436cb7cde832e7f93875" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 45549 } } peers { permanent_uuid: "c943855e209e48fb8496efe28c4d3859" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 38750 } } peers { permanent_uuid: "547f9023097149b9ba85f8c1bbe37faf" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 51298 } } I0529 14:48:42.518095 64003 raft_consensus.cc:213] T 7efc7c9a12544f9a94b4ddba475e889d P c943855e209e48fb8496efe28c4d3859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0529 14:48:42.518117 64005 tablet_bootstrap.cc:311] Tablet 7efc7c9a12544f9a94b4ddba475e889d: No bootstrap required, opened a new log I0529 14:48:42.520999 64005 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.030s user 0.022s sys 0.006s I0529 14:48:42.522208 64006 raft_consensus.cc:182] T 7efc7c9a12544f9a94b4ddba475e889d P 547f9023097149b9ba85f8c1bbe37faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "c7db368a9e6f436cb7cde832e7f93875" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 45549 } } peers { permanent_uuid: "c943855e209e48fb8496efe28c4d3859" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 38750 } } peers { permanent_uuid: "547f9023097149b9ba85f8c1bbe37faf" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 51298 } } I0529 14:48:42.523522 64006 raft_consensus.cc:213] T 7efc7c9a12544f9a94b4ddba475e889d P 547f9023097149b9ba85f8c1bbe37faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0529 14:48:42.518334 64003 raft_consensus.cc:360] T 7efc7c9a12544f9a94b4ddba475e889d P c943855e209e48fb8496efe28c4d3859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c943855e209e48fb8496efe28c4d3859, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0529 14:48:42.530123 64003 consensus_queue.cc:117] T 7efc7c9a12544f9a94b4ddba475e889d P c943855e209e48fb8496efe28c4d3859 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:42.529727 64005 raft_consensus.cc:182] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "c7db368a9e6f436cb7cde832e7f93875" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 45549 } } peers { permanent_uuid: "c943855e209e48fb8496efe28c4d3859" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 38750 } } peers { permanent_uuid: "547f9023097149b9ba85f8c1bbe37faf" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 51298 } } I0529 14:48:42.531894 64005 raft_consensus.cc:213] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0529 14:48:42.533795 64003 ts_tablet_manager.cc:310] Time spent starting tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.024s user 0.019s sys 0.004s I0529 14:48:42.523751 64006 raft_consensus.cc:360] T 7efc7c9a12544f9a94b4ddba475e889d P 547f9023097149b9ba85f8c1bbe37faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547f9023097149b9ba85f8c1bbe37faf, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0529 14:48:42.536142 64006 consensus_queue.cc:117] T 7efc7c9a12544f9a94b4ddba475e889d P 547f9023097149b9ba85f8c1bbe37faf [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:42.539587 64006 ts_tablet_manager.cc:310] Time spent starting tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.023s user 0.020s sys 0.002s I0529 14:48:42.532234 64005 raft_consensus.cc:360] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7db368a9e6f436cb7cde832e7f93875, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0529 14:48:42.548277 64005 consensus_queue.cc:117] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:42.552546 64005 ts_tablet_manager.cc:310] Time spent starting tablet 7efc7c9a12544f9a94b4ddba475e889d: real 0.031s user 0.023s sys 0.007s I0529 14:48:42.561877 62863 external_mini_cluster.cc:285] 3 TS(s) registered with Master I0529 14:48:42.744304 64017 raft_consensus.cc:310] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 0 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:42.744783 64017 raft_consensus.cc:1528] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 0 FOLLOWER]: Advancing to term 1 I0529 14:48:42.747161 64017 raft_consensus.cc:1478] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 1 FOLLOWER]: Snoozing failure detection for an additional: 1.568s I0529 14:48:42.747366 64017 raft_consensus.cc:273] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 1 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "c7db368a9e6f436cb7cde832e7f93875" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 45549 } } peers { permanent_uuid: "c943855e209e48fb8496efe28c4d3859" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 38750 } } peers { permanent_uuid: "547f9023097149b9ba85f8c1bbe37faf" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 51298 } } I0529 14:48:42.760845 64017 leader_election.cc:194] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [CANDIDATE]: Term 1 election: Requesting vote from peer c943855e209e48fb8496efe28c4d3859 I0529 14:48:42.761468 64017 leader_election.cc:194] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [CANDIDATE]: Term 1 election: Requesting vote from peer 547f9023097149b9ba85f8c1bbe37faf I0529 14:48:42.768038 63237 raft_consensus.cc:1528] T 7efc7c9a12544f9a94b4ddba475e889d P c943855e209e48fb8496efe28c4d3859 [term 0 FOLLOWER]: Advancing to term 1 I0529 14:48:42.770107 63405 raft_consensus.cc:1528] T 7efc7c9a12544f9a94b4ddba475e889d P 547f9023097149b9ba85f8c1bbe37faf [term 0 FOLLOWER]: Advancing to term 1 I0529 14:48:42.770220 63237 raft_consensus.cc:1478] T 7efc7c9a12544f9a94b4ddba475e889d P c943855e209e48fb8496efe28c4d3859 [term 1 FOLLOWER]: Snoozing failure detection for an additional: 1.610s I0529 14:48:42.772600 63405 raft_consensus.cc:1478] T 7efc7c9a12544f9a94b4ddba475e889d P 547f9023097149b9ba85f8c1bbe37faf [term 1 FOLLOWER]: Snoozing failure detection for an additional: 1.530s I0529 14:48:42.776906 63237 raft_consensus.cc:1261] T 7efc7c9a12544f9a94b4ddba475e889d P c943855e209e48fb8496efe28c4d3859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7db368a9e6f436cb7cde832e7f93875 in term 1. I0529 14:48:42.778255 63495 leader_election.cc:313] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [CANDIDATE]: Term 1 election: Vote granted by peer c943855e209e48fb8496efe28c4d3859 I0529 14:48:42.778594 63495 leader_election.cc:214] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0529 14:48:42.779639 64027 raft_consensus.cc:1478] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 1 FOLLOWER]: Snoozing failure detection for an additional: 1.642s I0529 14:48:42.779959 64027 raft_consensus.cc:1396] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 1 FOLLOWER]: Leader election won for term 1 I0529 14:48:42.782376 64027 raft_consensus.cc:323] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 1 LEADER]: Becoming Leader. State: Replica: c7db368a9e6f436cb7cde832e7f93875, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0529 14:48:42.783721 64027 consensus_queue.cc:109] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [LEADER]: queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER I0529 14:48:42.793303 63405 raft_consensus.cc:1261] T 7efc7c9a12544f9a94b4ddba475e889d P 547f9023097149b9ba85f8c1bbe37faf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7db368a9e6f436cb7cde832e7f93875 in term 1. I0529 14:48:42.794623 63496 leader_election.cc:313] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [CANDIDATE]: Term 1 election: Vote granted by peer 547f9023097149b9ba85f8c1bbe37faf I0529 14:48:42.796350 63063 catalog_manager.cc:1393] Tablet: 7efc7c9a12544f9a94b4ddba475e889d reported consensus state change. New consensus state: current_term: 1 leader_uuid: "c7db368a9e6f436cb7cde832e7f93875" quorum { opid_index: -1 local: false peers { permanent_uuid: "c7db368a9e6f436cb7cde832e7f93875" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 45549 } } peers { permanent_uuid: "c943855e209e48fb8496efe28c4d3859" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 38750 } } peers { permanent_uuid: "547f9023097149b9ba85f8c1bbe37faf" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 51298 } } } W0529 14:48:42.798599 64052 raft_consensus_state.cc:430] T 7efc7c9a12544f9a94b4ddba475e889d P c7db368a9e6f436cb7cde832e7f93875 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 W0529 14:48:42.919867 64106 reactor.cc:200] Client call kudu.consensus.ConsensusService.GetNodeInstance has no timeout set for connection id: {remote=127.245.143.2:45549, user_credentials={real_user=todd, eff_user=}} I0529 14:48:42.929756 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-master with pid 63007 I0529 14:48:42.988562 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server with pid 63094 I0529 14:48:43.041240 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server with pid 63315 I0529 14:48:43.090335 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server with pid 63472 [ OK ] RaftConsensusITest.TestGetPermanentUuid (5598 ms) [ RUN ] RaftConsensusITest.InsertWithCrashyNodes I0529 14:48:43.213815 62863 test_util.cc:118] Using random seed: 1889281987 I0529 14:48:43.221853 62863 ts_itest-base.h:72] Starting cluster with: I0529 14:48:43.221957 62863 ts_itest-base.h:73] -------------- I0529 14:48:43.222024 62863 ts_itest-base.h:74] 3 tablet servers I0529 14:48:43.222091 62863 ts_itest-base.h:75] 3 replicas per TS I0529 14:48:43.222156 62863 ts_itest-base.h:76] -------------- I0529 14:48:43.225319 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-master kudu-master --master_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/master --master_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --master_web_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/master --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 kudu-master: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-master: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:43.425082 64190 master_main.cc:27] Initializing master server... I0529 14:48:43.442538 64190 server_base.cc:136] Could not load existing FS layout: Not found: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/master/instance: No such file or directory (error 2) I0529 14:48:43.442850 64190 server_base.cc:137] Creating new FS layout I0529 14:48:43.531324 64190 fs_manager.cc:270] Generated new instance metadata in path /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/master/instance: uuid: "4fe3212b515b439087cb024cd88dbb92" format_stamp: "Formatted at 2015-05-29 21:48:43 on e1205.halxg.cloudera.com" I0529 14:48:43.610024 64190 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/master uuid: "4fe3212b515b439087cb024cd88dbb92" format_stamp: "Formatted at 2015-05-29 21:48:43 on e1205.halxg.cloudera.com" I0529 14:48:43.683787 64190 master_main.cc:30] Starting Master server... I0529 14:48:43.684270 64190 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:43.757874 64190 rpc_server.cc:110] RPC server started. Bound to: 127.0.0.1:59882 I0529 14:48:43.759819 64190 webserver.cc:108] Starting webserver on localhost:0 I0529 14:48:43.759938 64190 webserver.cc:117] Document root disabled I0529 14:48:43.761059 64190 net_util.cc:149] Address 127.0.0.1:0 for localhost:0 duplicates an earlier resolved entry. I0529 14:48:43.762802 64190 webserver.cc:200] Webserver started. Bound to: http://127.0.0.1:58221/ I0529 14:48:43.767204 64190 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/master/info.pb I0529 14:48:43.773103 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-master as pid 64190 I0529 14:48:43.775157 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.245.143.0:0 --local_ip_for_outbound_sockets=127.245.143.0 --webserver_interface=127.245.143.0 --tablet_server_web_port=0 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --fault_crash_on_leader_request_fraction=0.05 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:43.860121 64293 tablet_bootstrap.cc:311] Tablet 00000000000000000000000000000000: Bootstrap starting. I0529 14:48:43.864945 64293 tablet_bootstrap.cc:472] Time spent opening tablet 00000000000000000000000000000000: real 0.002s user 0.002s sys 0.000s I0529 14:48:43.866001 64293 tablet_bootstrap.cc:408] No blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I0529 14:48:43.870159 64293 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:43.877967 64293 tablet_bootstrap.cc:311] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I0529 14:48:43.881909 64293 local_consensus.cc:55] T 00000000000000000000000000000000 P 4fe3212b515b439087cb024cd88dbb92: Starting LocalConsensus... I0529 14:48:43.883898 64293 sys_catalog.cc:195] T 00000000000000000000000000000000 P 4fe3212b515b439087cb024cd88dbb92 [sys.catalog]: SysCatalogTable state changed. New quorum config:opid_index: -1 local: true peers { permanent_uuid: "4fe3212b515b439087cb024cd88dbb92" member_type: VOTER } I0529 14:48:43.884179 64293 sys_catalog.cc:198] T 00000000000000000000000000000000 P 4fe3212b515b439087cb024cd88dbb92 [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER I0529 14:48:43.889253 64293 sys_catalog.cc:263] T 00000000000000000000000000000000 P 4fe3212b515b439087cb024cd88dbb92 [sys.catalog]: configured and running, proceeding with master startup. I0529 14:48:43.889539 64301 catalog_manager.cc:451] Loading table and tablet metadata into memory... I0529 14:48:43.906854 64190 master_main.cc:33] Master server successfully started. I0529 14:48:43.975456 64294 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:43.991564 64294 server_base.cc:136] Could not load existing FS layout: Not found: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/instance: No such file or directory (error 2) I0529 14:48:43.991811 64294 server_base.cc:137] Creating new FS layout I0529 14:48:44.059900 64294 fs_manager.cc:270] Generated new instance metadata in path /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/instance: uuid: "3b7563857bb84965989d41eebdd82d5f" format_stamp: "Formatted at 2015-05-29 21:48:44 on e1205.halxg.cloudera.com" I0529 14:48:44.128244 64294 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 uuid: "3b7563857bb84965989d41eebdd82d5f" format_stamp: "Formatted at 2015-05-29 21:48:44 on e1205.halxg.cloudera.com" I0529 14:48:44.198639 64294 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:44.389926 64294 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.0:42480 I0529 14:48:44.393082 64294 webserver.cc:108] Starting webserver on 127.245.143.0:0 I0529 14:48:44.393347 64294 webserver.cc:117] Document root disabled I0529 14:48:44.395328 64294 webserver.cc:200] Webserver started. Bound to: http://127.245.143.0:41854/ I0529 14:48:44.397892 64294 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/info.pb I0529 14:48:44.399469 64294 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:44.400950 64294 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:44.404297 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 64294 I0529 14:48:44.405371 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.245.143.1:0 --local_ip_for_outbound_sockets=127.245.143.1 --webserver_interface=127.245.143.1 --tablet_server_web_port=0 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --fault_crash_on_leader_request_fraction=0.05 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:44.631067 64552 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:44.653131 64552 server_base.cc:136] Could not load existing FS layout: Not found: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/instance: No such file or directory (error 2) I0529 14:48:44.653386 64552 server_base.cc:137] Creating new FS layout I0529 14:48:44.737598 64552 fs_manager.cc:270] Generated new instance metadata in path /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/instance: uuid: "aa0d62233d0a498886326226b4827f51" format_stamp: "Formatted at 2015-05-29 21:48:44 on e1205.halxg.cloudera.com" I0529 14:48:44.808182 64552 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 uuid: "aa0d62233d0a498886326226b4827f51" format_stamp: "Formatted at 2015-05-29 21:48:44 on e1205.halxg.cloudera.com" I0529 14:48:44.883230 64552 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:45.036242 64548 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:45.036861 64548 heartbeater.cc:312] Registering TS with master... I0529 14:48:45.037500 64548 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:45.041033 64257 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" instance_seqno: 1432936124338678 } with Master I0529 14:48:45.078737 64552 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.1:43880 I0529 14:48:45.080854 64552 webserver.cc:108] Starting webserver on 127.245.143.1:0 I0529 14:48:45.080976 64552 webserver.cc:117] Document root disabled I0529 14:48:45.082882 64552 webserver.cc:200] Webserver started. Bound to: http://127.245.143.1:51798/ I0529 14:48:45.084321 64552 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/info.pb I0529 14:48:45.086004 64552 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:45.087551 64552 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:45.089042 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 64552 I0529 14:48:45.090467 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.245.143.2:0 --local_ip_for_outbound_sockets=127.245.143.2 --webserver_interface=127.245.143.2 --tablet_server_web_port=0 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --fault_crash_on_leader_request_fraction=0.05 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:45.340430 64815 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:45.360236 64815 server_base.cc:136] Could not load existing FS layout: Not found: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/instance: No such file or directory (error 2) I0529 14:48:45.360540 64815 server_base.cc:137] Creating new FS layout I0529 14:48:45.550279 64815 fs_manager.cc:270] Generated new instance metadata in path /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/instance: uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" format_stamp: "Formatted at 2015-05-29 21:48:45 on e1205.halxg.cloudera.com" I0529 14:48:45.629920 64815 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" format_stamp: "Formatted at 2015-05-29 21:48:45 on e1205.halxg.cloudera.com" I0529 14:48:45.697268 64815 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:45.703552 64812 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:45.704157 64812 heartbeater.cc:312] Registering TS with master... I0529 14:48:45.704668 64812 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:45.707609 64259 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "aa0d62233d0a498886326226b4827f51" instance_seqno: 1432936125036765 } with Master I0529 14:48:45.899737 64815 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.2:53754 I0529 14:48:45.902143 64815 webserver.cc:108] Starting webserver on 127.245.143.2:0 I0529 14:48:45.902403 64815 webserver.cc:117] Document root disabled I0529 14:48:45.904381 64815 webserver.cc:200] Webserver started. Bound to: http://127.245.143.2:34287/ I0529 14:48:45.905714 64815 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/info.pb I0529 14:48:45.907835 64815 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:45.909370 64815 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:45.913365 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 64815 I0529 14:48:46.580762 65116 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:46.581271 65116 heartbeater.cc:312] Registering TS with master... I0529 14:48:46.581673 65116 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:46.583700 64268 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" instance_seqno: 1432936125864990 } with Master I0529 14:48:46.584949 62863 external_mini_cluster.cc:285] 3 TS(s) registered with Master W0529 14:48:46.585541 64189 reactor.cc:200] Client call kudu.master.MasterService.ListTabletServers has no timeout set for connection id: {remote=127.0.0.1:59882, user_credentials={real_user=todd, eff_user=}} I0529 14:48:46.607949 64258 catalog_manager.cc:665] CreateTable from {real_user=todd, eff_user=} at 127.0.0.1:60261: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I0529 14:48:47.441428 64459 tablet_service.cc:420] Processing CreateTablet for tablet 05b4fe7d1e414fb79249c14e6ae31236 (table=test-workload [id=cd43a9f1ab3241359f697d234129b17d]), range=["", ""] I0529 14:48:47.441465 64723 tablet_service.cc:420] Processing CreateTablet for tablet 05b4fe7d1e414fb79249c14e6ae31236 (table=test-workload [id=cd43a9f1ab3241359f697d234129b17d]), range=["", ""] I0529 14:48:47.441905 65025 tablet_service.cc:420] Processing CreateTablet for tablet 05b4fe7d1e414fb79249c14e6ae31236 (table=test-workload [id=cd43a9f1ab3241359f697d234129b17d]), range=["", ""] I0529 14:48:47.528290 65025 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:47.528290 64459 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:47.528491 64723 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:47.530799 65405 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:47.530925 65406 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:47.531050 65407 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:47.536711 65407 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. I0529 14:48:47.538348 65406 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. I0529 14:48:47.538367 65405 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. I0529 14:48:47.542970 65407 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.002s user 0.002s sys 0.000s I0529 14:48:47.543835 65407 tablet_bootstrap.cc:408] No blocks or log segments found for tablet: 05b4fe7d1e414fb79249c14e6ae31236 creating new one. I0529 14:48:47.547207 65406 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.003s user 0.003s sys 0.000s I0529 14:48:47.547214 65405 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.003s user 0.002s sys 0.000s I0529 14:48:47.547652 65407 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:47.547770 65406 tablet_bootstrap.cc:408] No blocks or log segments found for tablet: 05b4fe7d1e414fb79249c14e6ae31236 creating new one. I0529 14:48:47.547794 65405 tablet_bootstrap.cc:408] No blocks or log segments found for tablet: 05b4fe7d1e414fb79249c14e6ae31236 creating new one. I0529 14:48:47.550312 65406 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:47.550315 65405 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:47.553669 65407 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: No bootstrap required, opened a new log I0529 14:48:47.554087 65407 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 88ms of latency in Log::Sync() I0529 14:48:47.559262 65405 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: No bootstrap required, opened a new log I0529 14:48:47.559327 65406 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: No bootstrap required, opened a new log I0529 14:48:47.559643 65405 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 129ms of latency in Log::Sync() I0529 14:48:47.563312 65406 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.032s user 0.024s sys 0.005s I0529 14:48:47.573976 65406 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:47.576333 65406 raft_consensus.cc:213] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0529 14:48:47.576884 65406 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8655f18f7e4c6691f4c12ebb22fd06, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0529 14:48:47.587615 65406 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:47.591276 65406 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.028s user 0.020s sys 0.006s I0529 14:48:47.591378 65438 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 0 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:47.591842 65438 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 0 FOLLOWER]: Advancing to term 1 I0529 14:48:47.595571 65438 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Snoozing failure detection for an additional: 0.316s I0529 14:48:47.595887 65438 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:47.611829 65438 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 1 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:47.612833 65438 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 1 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f W0529 14:48:47.622030 64997 leader_election.cc:247] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aa0d62233d0a498886326226b4827f51: Illegal state: Tablet not RUNNING: BOOTSTRAPPING W0529 14:48:47.624163 64995 leader_election.cc:247] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3b7563857bb84965989d41eebdd82d5f: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0529 14:48:47.624754 64995 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0529 14:48:47.626453 65451 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Snoozing failure detection for an additional: 0.310s I0529 14:48:47.627243 65451 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I0529 14:48:47.646575 65407 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.115s user 0.021s sys 0.006s I0529 14:48:47.657522 65407 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:47.659303 65407 raft_consensus.cc:213] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0529 14:48:47.659557 65407 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0529 14:48:47.672150 65407 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:47.669803 65494 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 0 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:47.673149 65494 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 0 FOLLOWER]: Advancing to term 1 I0529 14:48:47.676777 65494 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 1 FOLLOWER]: Snoozing failure detection for an additional: 0.324s I0529 14:48:47.677167 65407 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.030s user 0.023s sys 0.005s I0529 14:48:47.677130 65494 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 1 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:47.684245 65494 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 1 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:47.684976 65494 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 1 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f I0529 14:48:47.692769 65047 raft_consensus.cc:1215] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 in current term 1: Already voted for candidate 1f8655f18f7e4c6691f4c12ebb22fd06 in this term. I0529 14:48:47.692875 65405 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.162s user 0.022s sys 0.008s I0529 14:48:47.694917 64693 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 1 election: Vote denied by peer 1f8655f18f7e4c6691f4c12ebb22fd06. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 in current term 1: Already voted for candidate 1f8655f18f7e4c6691f4c12ebb22fd06 in this term. W0529 14:48:47.703045 64693 leader_election.cc:247] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3b7563857bb84965989d41eebdd82d5f: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0529 14:48:47.703670 64693 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0529 14:48:47.706601 65507 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 1 FOLLOWER]: Snoozing failure detection for an additional: 0.304s I0529 14:48:47.706851 65507 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I0529 14:48:47.707309 65405 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:47.709400 65405 raft_consensus.cc:213] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0529 14:48:47.709751 65405 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7563857bb84965989d41eebdd82d5f, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0529 14:48:47.726780 65405 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:47.731307 65405 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.038s user 0.029s sys 0.008s I0529 14:48:47.768112 65520 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 0 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:47.768733 65520 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 0 FOLLOWER]: Advancing to term 1 I0529 14:48:47.772222 65520 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 1 FOLLOWER]: Snoozing failure detection for an additional: 0.324s I0529 14:48:47.772608 65520 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 1 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:47.784147 65520 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 1 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:47.784744 65520 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 1 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:47.790292 65048 raft_consensus.cc:1215] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f in current term 1: Already voted for candidate 1f8655f18f7e4c6691f4c12ebb22fd06 in this term. I0529 14:48:47.791754 64428 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 1 election: Vote denied by peer 1f8655f18f7e4c6691f4c12ebb22fd06. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f in current term 1: Already voted for candidate 1f8655f18f7e4c6691f4c12ebb22fd06 in this term. I0529 14:48:47.792116 64745 raft_consensus.cc:1215] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f in current term 1: Already voted for candidate aa0d62233d0a498886326226b4827f51 in this term. I0529 14:48:47.793283 64430 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 1 election: Vote denied by peer aa0d62233d0a498886326226b4827f51. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f in current term 1: Already voted for candidate aa0d62233d0a498886326226b4827f51 in this term. I0529 14:48:47.793537 64430 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0529 14:48:47.794392 65523 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 1 FOLLOWER]: Snoozing failure detection for an additional: 0.329s I0529 14:48:47.794675 65523 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: None given I0529 14:48:48.251245 65438 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:48.251696 65438 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 1 FOLLOWER]: Advancing to term 2 I0529 14:48:48.253640 65438 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 FOLLOWER]: Snoozing failure detection for an additional: 0.346s I0529 14:48:48.253832 65438 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:48.268190 65438 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 2 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:48.268978 65438 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 2 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f I0529 14:48:48.270571 64746 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 1 FOLLOWER]: Advancing to term 2 I0529 14:48:48.270969 64482 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 1 FOLLOWER]: Advancing to term 2 I0529 14:48:48.273025 64746 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 2 FOLLOWER]: Snoozing failure detection for an additional: 0.321s I0529 14:48:48.273257 64482 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 2 FOLLOWER]: Snoozing failure detection for an additional: 0.324s I0529 14:48:48.281890 64482 raft_consensus.cc:1261] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f8655f18f7e4c6691f4c12ebb22fd06 in term 2. I0529 14:48:48.284265 64995 leader_election.cc:313] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 2 election: Vote granted by peer 3b7563857bb84965989d41eebdd82d5f I0529 14:48:48.284492 64995 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0529 14:48:48.287503 442 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 FOLLOWER]: Snoozing failure detection for an additional: 0.325s I0529 14:48:48.288079 442 raft_consensus.cc:1396] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 FOLLOWER]: Leader election won for term 2 I0529 14:48:48.290218 442 raft_consensus.cc:323] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 LEADER]: Becoming Leader. State: Replica: 1f8655f18f7e4c6691f4c12ebb22fd06, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0529 14:48:48.290952 442 consensus_queue.cc:109] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 2, Majority size: 2, State: 1, Mode: LEADER I0529 14:48:48.291013 64746 raft_consensus.cc:1261] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f8655f18f7e4c6691f4c12ebb22fd06 in term 2. I0529 14:48:48.291873 64997 leader_election.cc:313] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 2 election: Vote granted by peer aa0d62233d0a498886326226b4827f51 I0529 14:48:48.300215 65429 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 12ms of latency in Log::Sync() I0529 14:48:48.302937 64257 catalog_manager.cc:1393] Tablet: 05b4fe7d1e414fb79249c14e6ae31236 reported consensus state change. New consensus state: current_term: 2 leader_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" quorum { opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } } W0529 14:48:48.316202 465 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 2 I0529 14:48:48.342090 65429 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 114ms of latency in Log::Sync() W0529 14:48:48.342687 445 fault_injection.cc:19] FAULT INJECTION ENABLED! W0529 14:48:48.342936 445 fault_injection.cc:20] THIS SERVER MAY CRASH! I0529 14:48:48.346410 64747 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 2 FOLLOWER]: Refusing update from remote peer 1f8655f18f7e4c6691f4c12ebb22fd06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0529 14:48:48.346529 64484 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 2 FOLLOWER]: Refusing update from remote peer 1f8655f18f7e4c6691f4c12ebb22fd06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0529 14:48:48.348459 445 consensus_queue.cc:420] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: Connected to new peer: Peer: 3b7563857bb84965989d41eebdd82d5f, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR I0529 14:48:48.349455 442 consensus_queue.cc:420] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: Connected to new peer: Peer: aa0d62233d0a498886326226b4827f51, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR I0529 14:48:48.363679 65428 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 97ms of latency in Log::Sync() I0529 14:48:48.365461 65424 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 27ms of latency in Log::Sync() I0529 14:48:48.397552 465 raft_consensus_state.cc:423] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 2 index: 1 I0529 14:48:48.461344 65429 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 82ms of latency in Log::Sync() I0529 14:48:48.476958 65428 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 15ms of latency in Log::Sync() I0529 14:48:48.478806 65424 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 33ms of latency in Log::Sync() I0529 14:48:48.511960 65428 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 42ms of latency in Log::Sync() I0529 14:48:48.526329 65424 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 69ms of latency in Log::Sync() I0529 14:48:48.567668 65429 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 55ms of latency in Log::Sync() I0529 14:48:48.573035 65428 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 27ms of latency in Log::Sync() I0529 14:48:48.611337 65428 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 104ms of latency in Log::Sync() F0529 14:48:49.225992 442 fault_injection.cc:36] Injected fault: FLAGS_fault_crash_on_leader_request_fraction *** Check failure stack trace: *** @ 0x7f42f31f4bca google::LogMessage::Fail() @ 0x7f42f31f4496 google::LogMessage::SendToLog() @ 0x7f42f31f48ee google::LogMessage::Flush() @ 0x7f42f31f808c google::LogMessageFatal::~LogMessageFatal() @ 0x7f42ec0ee45f kudu::fault_injection::DoMaybeFault() @ 0x7f42f0d1589c kudu::fault_injection::MaybeFault() @ 0x7f42f0d129da kudu::consensus::Peer::SendNextRequest() @ 0x7f42f0d17f38 kudu::internal::RunnableAdapter<>::Run() @ 0x7f42f0d17e7f kudu::internal::InvokeHelper<>::MakeItSo() @ 0x7f42f0d17db4 kudu::internal::Invoker<>::Run() @ 0x7f42f0d357be kudu::Callback<>::Run() @ 0x7f42f0d364fe boost::_mfi::cmf0<>::operator()() @ 0x7f42f0d3646f boost::_bi::list1<>::operator()<>() @ 0x7f42f0d36424 boost::_bi::bind_t<>::operator()() @ 0x7f42f0d361b2 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7f42efe483e2 boost::function0<>::operator()() @ 0x7f42ec16b71e kudu::FunctionRunnable::Run() @ 0x7f42ec168551 kudu::ThreadPool::DispatchThread() @ 0x7f42ec169ca6 boost::_mfi::mf1<>::operator()() W0529 14:48:49.311312 496 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 46) took 807ms (client timeout 999). W0529 14:48:49.312391 497 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 47) took 802ms (client timeout 999). W0529 14:48:49.312809 490 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 48) took 802ms (client timeout 999). W0529 14:48:49.313978 505 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 50) took 803ms (client timeout 999). @ 0x7f42ec169c0c boost::_bi::list2<>::operator()<>() W0529 14:48:49.314617 491 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 49) took 804ms (client timeout 999). W0529 14:48:49.316269 497 inbound_call.cc:193] Trace: 0529 14:48:48.509174 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.509506 (+ 332us) service_pool.cc:158] Handling call 0529 14:48:49.311813 (+802307us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.510447 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.510739 (+ 292us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.510775 (+ 36us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.510801 (+ 26us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.511024 (+ 223us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.511163 (+ 139us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.511186 (+ 23us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.511252 (+ 66us) write_transaction.cc:84] Start() 0529 14:48:48.511374 (+ 122us) write_transaction.cc:86] Timestamp: L: 18 0529 14:48:48.511996 (+ 622us) log.cc:374] Serialized 170 byte log entry 0529 14:48:48.600162 (+ 88166us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.310823 (+710661us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.311015 (+ 192us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.311140 (+ 125us) write_transaction.cc:225] Released schema lock 0529 14:48:49.311542 (+ 402us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.311609 (+ 67us) write_transaction.cc:142] FINISH: making edits visible W0529 14:48:49.316269 496 inbound_call.cc:193] Trace: 0529 14:48:48.503549 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.504127 (+ 578us) service_pool.cc:158] Handling call 0529 14:48:49.310901 (+806774us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.507945 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.508249 (+ 304us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.508287 (+ 38us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.508305 (+ 18us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.508522 (+ 217us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.508642 (+ 120us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.508654 (+ 12us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.508699 (+ 45us) write_transaction.cc:84] Start() 0529 14:48:48.508772 (+ 73us) write_transaction.cc:86] Timestamp: L: 17 0529 14:48:48.509194 (+ 422us) log.cc:374] Serialized 170 byte log entry 0529 14:48:48.599895 (+ 90701us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.309099 (+709204us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.309477 (+ 378us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.309610 (+ 133us) write_transaction.cc:225] Released schema lock 0529 14:48:49.310429 (+ 819us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.310519 (+ 90us) write_transaction.cc:142] FINISH: making edits visible W0529 14:48:49.316890 490 inbound_call.cc:193] Trace: 0529 14:48:48.509555 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.510031 (+ 476us) service_pool.cc:158] Handling call 0529 14:48:49.312307 (+802276us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.512251 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.512449 (+ 198us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.512470 (+ 21us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.512483 (+ 13us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.512627 (+ 144us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.512719 (+ 92us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.512733 (+ 14us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.512772 (+ 39us) write_transaction.cc:84] Start() 0529 14:48:48.512847 (+ 75us) write_transaction.cc:86] Timestamp: L: 19 0529 14:48:48.513288 (+ 441us) log.cc:374] Serialized 170 byte log entry 0529 14:48:48.600801 (+ 87513us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.310759 (+709958us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.311090 (+ 331us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.311212 (+ 122us) write_transaction.cc:225] Released schema lock 0529 14:48:49.311773 (+ 561us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.311884 (+ 111us) write_transaction.cc:142] FINISH: making edits visible W0529 14:48:49.317337 505 inbound_call.cc:193] Trace: 0529 14:48:48.510279 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.510552 (+ 273us) service_pool.cc:158] Handling call 0529 14:48:49.313502 (+802950us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.514681 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.514864 (+ 183us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.514886 (+ 22us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.514896 (+ 10us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.515035 (+ 139us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.515126 (+ 91us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.515137 (+ 11us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.515179 (+ 42us) write_transaction.cc:84] Start() 0529 14:48:48.515254 (+ 75us) write_transaction.cc:86] Timestamp: L: 21 0529 14:48:48.516215 (+ 961us) log.cc:374] Serialized 170 byte log entry 0529 14:48:49.311697 (+795482us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.312163 (+ 466us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.312373 (+ 210us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.312517 (+ 144us) write_transaction.cc:225] Released schema lock 0529 14:48:49.313093 (+ 576us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.313167 (+ 74us) write_transaction.cc:142] FINISH: making edits visible W0529 14:48:49.317435 711 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 51) took 753ms (client timeout 999). W0529 14:48:49.320569 491 inbound_call.cc:193] Trace: 0529 14:48:48.509913 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.510216 (+ 303us) service_pool.cc:158] Handling call 0529 14:48:49.314200 (+803984us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.513527 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.513717 (+ 190us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.513738 (+ 21us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.513751 (+ 13us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.513883 (+ 132us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.513973 (+ 90us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.513984 (+ 11us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.514026 (+ 42us) write_transaction.cc:84] Start() 0529 14:48:48.514098 (+ 72us) write_transaction.cc:86] Timestamp: L: 20 0529 14:48:48.514451 (+ 353us) log.cc:374] Serialized 170 byte log entry 0529 14:48:49.311529 (+797078us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.312646 (+ 1117us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.312832 (+ 186us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.312946 (+ 114us) write_transaction.cc:225] Released schema lock 0529 14:48:49.313495 (+ 549us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.313585 (+ 90us) write_transaction.cc:142] FINISH: making edits visible @ 0x7f42ec169b94 boost::_bi::bind_t<>::operator()() W0529 14:48:49.321593 711 inbound_call.cc:193] Trace: 0529 14:48:48.563900 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.564024 (+ 124us) service_pool.cc:158] Handling call 0529 14:48:49.317015 (+752991us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.565260 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.565627 (+ 367us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.565667 (+ 40us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.565691 (+ 24us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.565955 (+ 264us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.566121 (+ 166us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.566147 (+ 26us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.566222 (+ 75us) write_transaction.cc:84] Start() 0529 14:48:48.566368 (+ 146us) write_transaction.cc:86] Timestamp: L: 22 0529 14:48:48.567356 (+ 988us) log.cc:374] Serialized 170 byte log entry 0529 14:48:49.315337 (+747981us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.316070 (+ 733us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.316244 (+ 174us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.316326 (+ 82us) write_transaction.cc:225] Released schema lock 0529 14:48:49.316795 (+ 469us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.316849 (+ 54us) write_transaction.cc:142] FINISH: making edits visible W0529 14:48:49.322226 497 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 52) took 758ms (client timeout 999). W0529 14:48:49.322679 496 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 53) took 758ms (client timeout 999). W0529 14:48:49.323642 490 inbound_call.cc:189] Call kudu.tserver.TabletServerService.Write from 127.245.143.2:45627 (request call id 54) took 758ms (client timeout 999). W0529 14:48:49.324192 497 inbound_call.cc:193] Trace: 0529 14:48:48.564169 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.564372 (+ 203us) service_pool.cc:158] Handling call 0529 14:48:49.322145 (+757773us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.567745 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.568473 (+ 728us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.568533 (+ 60us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.568564 (+ 31us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.568924 (+ 360us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.569158 (+ 234us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.569194 (+ 36us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.569300 (+ 106us) write_transaction.cc:84] Start() 0529 14:48:48.569497 (+ 197us) write_transaction.cc:86] Timestamp: L: 23 0529 14:48:48.570547 (+ 1050us) log.cc:374] Serialized 170 byte log entry 0529 14:48:49.320620 (+750073us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.321138 (+ 518us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.321296 (+ 158us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.321408 (+ 112us) write_transaction.cc:225] Released schema lock 0529 14:48:49.321860 (+ 452us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.321937 (+ 77us) write_transaction.cc:142] FINISH: making edits visible W0529 14:48:49.324550 496 inbound_call.cc:193] Trace: 0529 14:48:48.564453 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.565175 (+ 722us) service_pool.cc:158] Handling call 0529 14:48:49.322622 (+757447us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.571101 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.571881 (+ 780us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.571941 (+ 60us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.571981 (+ 40us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.572347 (+ 366us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.572575 (+ 228us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.572617 (+ 42us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.572718 (+ 101us) write_transaction.cc:84] Start() 0529 14:48:48.572907 (+ 189us) write_transaction.cc:86] Timestamp: L: 24 0529 14:48:48.573940 (+ 1033us) log.cc:374] Serialized 170 byte log entry 0529 14:48:49.321718 (+747778us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.321987 (+ 269us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.322075 (+ 88us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.322143 (+ 68us) write_transaction.cc:225] Released schema lock 0529 14:48:49.322419 (+ 276us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.322472 (+ 53us) write_transaction.cc:142] FINISH: making edits visible W0529 14:48:49.325323 490 inbound_call.cc:193] Trace: 0529 14:48:48.565122 (+ 0us) service_pool.cc:97] Inserting onto call queue 0529 14:48:48.565451 (+ 329us) service_pool.cc:158] Handling call 0529 14:48:49.323562 (+758111us) inbound_call.cc:110] Queueing success response Related trace: 0529 14:48:48.574493 (+ 0us) write_transaction.cc:52] PREPARE: Starting 0529 14:48:48.574900 (+ 407us) write_transaction.cc:218] Acquiring schema lock in shared mode 0529 14:48:48.574933 (+ 33us) write_transaction.cc:220] Acquired schema lock 0529 14:48:48.574951 (+ 18us) tablet.cc:234] PREPARE: Decoding operations 0529 14:48:48.575174 (+ 223us) tablet.cc:260] PREPARE: Acquiring locks for 1 operations 0529 14:48:48.575305 (+ 131us) tablet.cc:264] PREPARE: locks acquired 0529 14:48:48.575325 (+ 20us) write_transaction.cc:78] PREPARE: finished. 0529 14:48:48.575395 (+ 70us) write_transaction.cc:84] Start() 0529 14:48:48.575957 (+ 562us) write_transaction.cc:86] Timestamp: L: 25 0529 14:48:48.577080 (+ 1123us) log.cc:374] Serialized 170 byte log entry 0529 14:48:49.322324 (+745244us) write_transaction.cc:94] APPLY: Starting 0529 14:48:49.322748 (+ 424us) tablet_metrics.cc:197] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0529 14:48:49.322931 (+ 183us) write_transaction.cc:125] PRECOMMIT: Releasing row and schema locks 0529 14:48:49.323034 (+ 103us) write_transaction.cc:225] Released schema lock 0529 14:48:49.323285 (+ 251us) log.cc:374] Serialized 22 byte log entry 0529 14:48:49.323348 (+ 63us) write_transaction.cc:142] FINISH: making edits visible @ 0x7f42ec169992 boost::detail::function::void_function_obj_invoker0<>::invoke() I0529 14:48:49.330637 65424 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 9ms of latency in Log::Sync() I0529 14:48:49.331468 65429 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 54ms of latency in Log::Sync() @ 0x7f42efe483e2 boost::function0<>::operator()() @ 0x7f42ec15dd28 kudu::Thread::SuperviseThread() @ 0x7f42f39fbea2 __tsan::cur_thread_placeholder @ 0x7f42ed75e851 (unknown) @ 0x7f42ea31094d (unknown) @ (nil) (unknown) W0529 14:48:49.412935 64694 connection.cc:379] server Connection to 127.245.143.2:52916 recv error: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:49.413156 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) I0529 14:48:49.413832 62863 raft_consensus-itest.cc:805] TS 2 appears to have crashed. Restarting. W0529 14:48:49.413880 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:49.414131 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:49.414423 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:49.414751 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) I0529 14:48:49.414937 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.245.143.2:53754 --local_ip_for_outbound_sockets=127.245.143.2 --tablet_server_web_port=34287 --webserver_interface=127.245.143.2 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --fault_crash_on_leader_request_fraction=0.05 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 W0529 14:48:49.415030 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:49.415365 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:49.428028 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:49.428345 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:49.428676 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Recv() got EOF from remote (error 108) kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:49.458842 65520 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 2 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:49.459301 65520 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 2 FOLLOWER]: Advancing to term 3 W0529 14:48:49.460216 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.460489 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.460727 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.460958 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.461699 65520 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 3 FOLLOWER]: Snoozing failure detection for an additional: 0.325s I0529 14:48:49.461952 65520 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 3 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } W0529 14:48:49.463387 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.469362 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.469596 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.470690 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.473837 65520 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 3 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:49.474429 65520 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 3 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:49.476932 64762 raft_consensus.cc:1246] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for term 3 because replica is either leader or believes a valid leader to be alive. W0529 14:48:49.477134 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.477023 64428 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.477766 64430 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 3 election: Vote denied by peer aa0d62233d0a498886326226b4827f51. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for term 3 because replica is either leader or believes a valid leader to be alive. I0529 14:48:49.478003 64430 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. W0529 14:48:49.478415 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.479611 788 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 3 FOLLOWER]: Snoozing failure detection for an additional: 0.321s I0529 14:48:49.479812 788 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W0529 14:48:49.485725 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.485970 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.490350 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.495415 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.495712 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.500874 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.501843 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.505956 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.517616 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.531240 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.533427 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.540894 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.552371 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.552628 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.567101 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.568522 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.573057 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.587162 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.589298 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.598942 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.600078 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.609308 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.622966 784 tablet_server_main.cc:38] Initializing tablet server... W0529 14:48:49.631458 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.631808 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.632117 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.644917 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.651712 65494 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 2 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:49.653118 65494 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 2 FOLLOWER]: Advancing to term 3 I0529 14:48:49.657609 65494 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 3 FOLLOWER]: Snoozing failure detection for an additional: 0.307s I0529 14:48:49.657815 65494 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 3 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } W0529 14:48:49.661479 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.664831 65494 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 3 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:49.665174 65494 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 3 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f W0529 14:48:49.666292 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.667773 64693 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.668815 64494 raft_consensus.cc:1215] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 in current term 3: Already voted for candidate 3b7563857bb84965989d41eebdd82d5f in this term. I0529 14:48:49.666998 784 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" format_stamp: "Formatted at 2015-05-29 21:48:45 on e1205.halxg.cloudera.com" I0529 14:48:49.669570 64693 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 3 election: Vote denied by peer 3b7563857bb84965989d41eebdd82d5f. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 in current term 3: Already voted for candidate 3b7563857bb84965989d41eebdd82d5f in this term. I0529 14:48:49.669771 64693 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I0529 14:48:49.671861 801 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 3 FOLLOWER]: Snoozing failure detection for an additional: 0.315s I0529 14:48:49.672113 801 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: None given W0529 14:48:49.680778 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.686162 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.686466 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.687003 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.687333 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.702496 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.704082 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.738401 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.738679 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.739207 784 ts_tablet_manager.cc:261] Loading tablet 05b4fe7d1e414fb79249c14e6ae31236 W0529 14:48:49.740003 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:49.745174 784 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 W0529 14:48:49.765671 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.765950 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.766141 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.776906 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.787961 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.788185 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.788444 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.793550 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.812829 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.826838 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.848503 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.848758 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.850276 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.874517 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.874856 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.877346 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.882450 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.903251 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.929842 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.960409 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.980278 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:49.993111 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.011035 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.026660 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.059814 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.093451 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:50.113071 65520 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 3 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:50.113415 65520 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 3 FOLLOWER]: Advancing to term 4 I0529 14:48:50.115181 65520 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 4 FOLLOWER]: Snoozing failure detection for an additional: 0.326s I0529 14:48:50.115350 65520 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 4 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } W0529 14:48:50.117743 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:50.131815 65520 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 4 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:50.132340 65520 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 4 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:50.134310 64763 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 3 FOLLOWER]: Advancing to term 4 W0529 14:48:50.135082 64428 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:50.138042 64763 raft_consensus.cc:1232] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for term 4 because replica has last-logged OpId of term: 2 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 25. I0529 14:48:50.139065 64430 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 4 election: Vote denied by peer aa0d62233d0a498886326226b4827f51. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for term 4 because replica has last-logged OpId of term: 2 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 25. I0529 14:48:50.139289 64430 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I0529 14:48:50.140538 809 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:50.140894 970 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 4 FOLLOWER]: Snoozing failure detection for an additional: 0.314s I0529 14:48:50.141145 970 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: None given I0529 14:48:50.141469 784 tablet_server_main.cc:41] Starting tablet server... W0529 14:48:50.147807 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:50.149718 809 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. I0529 14:48:50.157744 809 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.004s user 0.003s sys 0.000s W0529 14:48:50.157935 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:50.158887 809 tablet_bootstrap.cc:549] Will attempt to recover log segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236/wal-000000001 to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 I0529 14:48:50.159129 809 tablet_bootstrap.cc:559] Moved log directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236 to recovery directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery W0529 14:48:50.161118 809 log_util.cc:149] Could not read footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001. Status: Not found: Footer not found. Footer magic doesn't match W0529 14:48:50.161332 809 log_reader.cc:138] Segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data I0529 14:48:50.169006 809 log_util.cc:203] Successfully rebuilt footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 (valid entries through byte offset 5916) I0529 14:48:50.172977 809 log.cc:277] Log is configured to *not* fsync() on all Append() calls W0529 14:48:50.183879 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.211292 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.212278 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.224831 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.244714 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.263051 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.282122 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.292801 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.293476 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:50.295213 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:50.303040 65494 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 4 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:50.303539 65494 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 4 FOLLOWER]: Advancing to term 5 I0529 14:48:50.320791 65494 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 FOLLOWER]: Snoozing failure detection for an additional: 0.329s I0529 14:48:50.321135 65494 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } W0529 14:48:50.328708 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:50.332329 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:50.336272 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:50.337573 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:50.338665 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:50.339938 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out I0529 14:48:50.346382 784 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.2:53754 I0529 14:48:50.348747 784 webserver.cc:108] Starting webserver on 127.245.143.2:34287 I0529 14:48:50.348871 784 webserver.cc:117] Document root disabled W0529 14:48:50.350188 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out I0529 14:48:50.351456 784 webserver.cc:200] Webserver started. Bound to: http://127.245.143.2:34287/ W0529 14:48:50.352860 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:50.366838 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out I0529 14:48:50.373430 65494 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 5 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:50.374299 65494 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 5 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f I0529 14:48:50.374732 784 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/info.pb I0529 14:48:50.376870 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 784 I0529 14:48:50.377491 64495 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 4 FOLLOWER]: Advancing to term 5 I0529 14:48:50.380736 64495 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 5 FOLLOWER]: Snoozing failure detection for an additional: 0.373s W0529 14:48:50.381970 64693 leader_election.cc:247] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 5 election: Tablet error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Illegal state: Tablet not RUNNING: BOOTSTRAPPING W0529 14:48:50.384805 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out I0529 14:48:50.431419 64495 raft_consensus.cc:1261] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0d62233d0a498886326226b4827f51 in term 5. I0529 14:48:50.432559 64693 leader_election.cc:313] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 5 election: Vote granted by peer 3b7563857bb84965989d41eebdd82d5f I0529 14:48:50.433099 64693 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I0529 14:48:50.434846 1100 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 FOLLOWER]: Snoozing failure detection for an additional: 0.336s I0529 14:48:50.435291 1100 raft_consensus.cc:1396] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 FOLLOWER]: Leader election won for term 5 I0529 14:48:50.437804 1100 raft_consensus.cc:323] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 LEADER]: Becoming Leader. State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: LEADER Watermarks: {Received: term: 2 index: 35 Committed: term: 2 index: 25} I0529 14:48:50.438724 1100 consensus_queue.cc:109] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [LEADER]: queue going to LEADER mode. State: All replicated op: 2.35, Majority replicated op: 2.25, Committed index: 2.25, Last appended: 2.35, Current term: 5, Majority size: 2, State: 1, Mode: LEADER I0529 14:48:50.444783 65424 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 66ms of latency in Log::Sync() I0529 14:48:50.448114 64264 catalog_manager.cc:1393] Tablet: 05b4fe7d1e414fb79249c14e6ae31236 reported consensus state change. New consensus state: current_term: 5 leader_uuid: "aa0d62233d0a498886326226b4827f51" quorum { opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } } W0529 14:48:50.452046 1103 fault_injection.cc:19] FAULT INJECTION ENABLED! W0529 14:48:50.452232 1103 fault_injection.cc:20] THIS SERVER MAY CRASH! I0529 14:48:50.455080 64496 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 5 FOLLOWER]: Refusing update from remote peer aa0d62233d0a498886326226b4827f51: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 5 index: 37. (index mismatch) W0529 14:48:50.456236 64693 consensus_peers.cc:216] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51: Couldn't send request to peer 1f8655f18f7e4c6691f4c12ebb22fd06 for tablet 05b4fe7d1e414fb79249c14e6ae31236 Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 1 times. I0529 14:48:50.457772 1103 consensus_queue.cc:420] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [LEADER]: Connected to new peer: Peer: 3b7563857bb84965989d41eebdd82d5f, Is new: false, Last received: 2.25, Next index: 26, Last known committed idx: 15, Last exchange result: ERROR F0529 14:48:50.497301 1103 fault_injection.cc:36] Injected fault: FLAGS_fault_crash_on_leader_request_fraction *** Check failure stack trace: *** W0529 14:48:50.497268 1122 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 35, Current term is: 5 @ 0x7f9e2e478bca google::LogMessage::Fail() @ 0x7f9e2e478496 google::LogMessage::SendToLog() @ 0x7f9e2e4788ee google::LogMessage::Flush() I0529 14:48:50.514953 65424 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 114ms of latency in Log::Sync() @ 0x7f9e2e47c08c google::LogMessageFatal::~LogMessageFatal() @ 0x7f9e2737245f kudu::fault_injection::DoMaybeFault() I0529 14:48:50.525228 1122 raft_consensus_state.cc:423] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 2 index: 25 New committed index is: term: 5 index: 36 @ 0x7f9e2bf9989c kudu::fault_injection::MaybeFault() @ 0x7f9e2bf969da kudu::consensus::Peer::SendNextRequest() @ 0x7f9e2bf975a9 kudu::consensus::Peer::DoProcessResponse() @ 0x7f9e2bf9b17e kudu::internal::RunnableAdapter<>::Run() @ 0x7f9e2bf9b0ec kudu::internal::InvokeHelper<>::MakeItSo() @ 0x7f9e2bf9b01b kudu::internal::Invoker<>::Run() @ 0x7f9e2bfb97be kudu::Callback<>::Run() @ 0x7f9e2bfba4fe boost::_mfi::cmf0<>::operator()() W0529 14:48:50.561745 64693 consensus_peers.cc:216] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51: Couldn't send request to peer 1f8655f18f7e4c6691f4c12ebb22fd06 for tablet 05b4fe7d1e414fb79249c14e6ae31236 Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. @ 0x7f9e2bfba46f boost::_bi::list1<>::operator()<>() @ 0x7f9e2bfba424 boost::_bi::bind_t<>::operator()() @ 0x7f9e2bfba1b2 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7f9e2b0cc3e2 boost::function0<>::operator()() @ 0x7f9e273ef71e kudu::FunctionRunnable::Run() @ 0x7f9e273ec551 kudu::ThreadPool::DispatchThread() @ 0x7f9e273edca6 boost::_mfi::mf1<>::operator()() @ 0x7f9e273edc0c boost::_bi::list2<>::operator()<>() @ 0x7f9e273edb94 boost::_bi::bind_t<>::operator()() @ 0x7f9e273ed992 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7f9e2b0cc3e2 boost::function0<>::operator()() @ 0x7f9e273e1d28 kudu::Thread::SuperviseThread() @ 0x7f9e2ec7fea2 __tsan::cur_thread_placeholder @ 0x7f9e289e2851 (unknown) I0529 14:48:50.615211 784 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:50.615474 809 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=25} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates @ 0x7f9e2559494d (unknown) I0529 14:48:50.618588 784 tablet_server_main.cc:44] Tablet server successfully started. @ (nil) (unknown) I0529 14:48:50.629763 1095 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:50.630441 1095 heartbeater.cc:312] Registering TS with master... I0529 14:48:50.630925 1095 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:50.633234 64259 ts_manager.cc:71] Re-registered known tablet server { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" instance_seqno: 1432936130301902 } with Master I0529 14:48:50.634785 809 tablet_bootstrap.cc:945] ReplayState: Previous OpId: 2.28, Committed OpId: 2.25, Pending Replicates: 3, Pending Commits: 0 I0529 14:48:50.634891 809 tablet_bootstrap.cc:945] Dumping REPLICATES: I0529 14:48:50.634961 809 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 26 } timestamp: 26 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\330\026\314R_\351\357T\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:50.635032 809 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 27 } timestamp: 27 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000F\022pu\346\355\277\020\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:50.635102 809 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 28 } timestamp: 28 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\311\311\2419\256\310\306,\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:50.640878 809 tablet_bootstrap.cc:577] Renamed old recovery dir from: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936130640759 I0529 14:48:50.641808 809 tablet_bootstrap.cc:585] Removed renamed recovery dir: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936130640759 I0529 14:48:50.642091 809 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap complete. I0529 14:48:50.642742 809 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 41ms of latency in Log::Sync() W0529 14:48:50.682253 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:50.682687 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:50.682965 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:50.683240 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:50.683517 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:50.683770 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:50.684031 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:50.684530 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) I0529 14:48:50.684563 62863 raft_consensus-itest.cc:805] TS 1 appears to have crashed. Restarting. W0529 14:48:50.684808 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) W0529 14:48:50.685120 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Recv() got EOF from remote (error 108) I0529 14:48:50.685379 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.245.143.1:43880 --local_ip_for_outbound_sockets=127.245.143.1 --tablet_server_web_port=51798 --webserver_interface=127.245.143.1 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --fault_crash_on_leader_request_fraction=0.05 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0529 14:48:50.687961 809 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.547s user 0.214s sys 0.074s I0529 14:48:50.696634 809 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 FOLLOWER]: Replica starting. Triggering 3 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:50.703784 809 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8655f18f7e4c6691f4c12ebb22fd06, State: 1, Role: FOLLOWER Watermarks: {Received: term: 2 index: 28 Committed: term: 2 index: 25} I0529 14:48:50.704504 809 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 2.28, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:50.711575 809 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.023s user 0.014s sys 0.007s W0529 14:48:50.716187 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.716696 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) W0529 14:48:50.723932 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.731750 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.733276 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.748735 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.762171 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.766898 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.782845 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.793372 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.794438 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.799327 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.818449 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:50.831918 65520 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 5 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:50.832397 65520 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 5 FOLLOWER]: Advancing to term 6 W0529 14:48:50.833068 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.833456 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.833765 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:50.834808 65520 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 FOLLOWER]: Snoozing failure detection for an additional: 0.365s W0529 14:48:50.835031 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:50.834980 65520 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } W0529 14:48:50.843691 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:50.848315 65520 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 6 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:50.848714 65520 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 6 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 W0529 14:48:50.851526 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.853665 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.854123 64430 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer aa0d62233d0a498886326226b4827f51: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:50.858001 1031 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 2 FOLLOWER]: Advancing to term 6 I0529 14:48:50.860312 1031 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 6 FOLLOWER]: Snoozing failure detection for an additional: 0.423s W0529 14:48:50.861455 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.861707 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:50.866770 1031 raft_consensus.cc:1261] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b7563857bb84965989d41eebdd82d5f in term 6. I0529 14:48:50.867931 64428 leader_election.cc:313] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 6 election: Vote granted by peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:50.868129 64428 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. I0529 14:48:50.868829 970 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 FOLLOWER]: Snoozing failure detection for an additional: 0.350s I0529 14:48:50.869163 970 raft_consensus.cc:1396] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 FOLLOWER]: Leader election won for term 6 I0529 14:48:50.871306 970 raft_consensus.cc:323] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 LEADER]: Becoming Leader. State: Replica: 3b7563857bb84965989d41eebdd82d5f, State: 1, Role: LEADER Watermarks: {Received: term: 5 index: 38 Committed: term: 2 index: 25} I0529 14:48:50.872143 970 consensus_queue.cc:109] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [LEADER]: queue going to LEADER mode. State: All replicated op: 5.38, Majority replicated op: 2.25, Committed index: 2.25, Last appended: 5.38, Current term: 6, Majority size: 2, State: 1, Mode: LEADER W0529 14:48:50.875133 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.876425 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:50.880921 1365 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 25, Current term is: 6 W0529 14:48:50.881969 1361 fault_injection.cc:19] FAULT INJECTION ENABLED! W0529 14:48:50.882056 1365 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 25, Current term is: 6 W0529 14:48:50.882184 1361 fault_injection.cc:20] THIS SERVER MAY CRASH! I0529 14:48:50.882354 64258 catalog_manager.cc:1393] Tablet: 05b4fe7d1e414fb79249c14e6ae31236 reported consensus state change. New consensus state: current_term: 6 leader_uuid: "3b7563857bb84965989d41eebdd82d5f" quorum { opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } } I0529 14:48:50.884436 65428 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 8ms of latency in Log::Sync() W0529 14:48:50.884624 64430 consensus_peers.cc:216] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f: Couldn't send request to peer aa0d62233d0a498886326226b4827f51 for tablet 05b4fe7d1e414fb79249c14e6ae31236 Status: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0529 14:48:50.884861 1032 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 6 FOLLOWER]: Refusing update from remote peer 3b7563857bb84965989d41eebdd82d5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 6 index: 40. (index mismatch) W0529 14:48:50.885907 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:50.886464 970 consensus_queue.cc:420] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [LEADER]: Connected to new peer: Peer: 1f8655f18f7e4c6691f4c12ebb22fd06, Is new: false, Last received: 2.28, Next index: 29, Last known committed idx: 25, Last exchange result: ERROR W0529 14:48:50.893075 1365 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 25, Current term is: 6 W0529 14:48:50.900008 1365 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 25, Current term is: 6 I0529 14:48:50.907076 65428 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 99ms of latency in Log::Sync() F0529 14:48:50.912734 1361 fault_injection.cc:36] Injected fault: FLAGS_fault_crash_on_leader_request_fraction *** Check failure stack trace: *** I0529 14:48:50.923544 1282 tablet_server_main.cc:38] Initializing tablet server... W0529 14:48:50.930147 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:50.964462 1282 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 uuid: "aa0d62233d0a498886326226b4827f51" format_stamp: "Formatted at 2015-05-29 21:48:44 on e1205.halxg.cloudera.com" W0529 14:48:50.980090 65198 connection.cc:379] client Connection to 127.245.143.0:42480 recv error: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.980936 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.981297 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.981600 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.981941 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.982343 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.982704 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.983171 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.983635 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.983903 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:50.984307 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:51.026571 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:51.037317 1282 ts_tablet_manager.cc:261] Loading tablet 05b4fe7d1e414fb79249c14e6ae31236 W0529 14:48:51.040119 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.040501 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.040837 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.041154 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.044013 1282 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 W0529 14:48:51.044406 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.049136 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.067859 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.084699 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.092363 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.100493 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.102682 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.102995 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.110489 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.121037 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.128017 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.133906 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.134207 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.135603 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.138335 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.143201 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.153791 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.167058 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.171120 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.178020 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.187762 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.188045 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.196038 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.214207 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.214485 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.216734 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:51.218536 1277 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 6 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:51.219022 1277 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 6 FOLLOWER]: Advancing to term 7 I0529 14:48:51.221424 1277 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 7 FOLLOWER]: Snoozing failure detection for an additional: 0.428s I0529 14:48:51.221688 1277 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 7 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:51.228229 1277 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 7 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:51.228929 1277 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 7 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f W0529 14:48:51.231163 806 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer aa0d62233d0a498886326226b4827f51: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.232131 804 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 3b7563857bb84965989d41eebdd82d5f: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.232261 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.232509 804 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. I0529 14:48:51.233664 1287 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 7 FOLLOWER]: Snoozing failure detection for an additional: 0.425s W0529 14:48:51.233724 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:51.233940 1287 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: None given W0529 14:48:51.246512 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.263262 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.264469 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.272683 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.278272 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.278507 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.279362 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.294596 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.303428 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.309989 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.325080 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.334745 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.334838 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.341096 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.343879 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.344364 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.347015 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.348285 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.352579 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.352846 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.359549 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.361634 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.362886 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.368860 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.369729 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.372122 65198 meta_cache.cc:498] GetTableLocations(test-workload, ) failed: Timed out: Call timed out W0529 14:48:51.374478 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.375058 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.378203 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.379534 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.379787 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.382676 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.383862 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.384115 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.385865 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.393173 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.394544 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.395637 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.407049 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.407923 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.410207 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.410433 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.412848 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.416491 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:51.426383 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.428644 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.428932 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.429260 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.429595 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:51.430611 1381 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 W0529 14:48:51.430876 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:51.432332 1282 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:51.440774 1381 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. W0529 14:48:51.443200 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.444319 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:51.448884 1381 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.003s user 0.003s sys 0.000s I0529 14:48:51.450433 1381 tablet_bootstrap.cc:549] Will attempt to recover log segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236/wal-000000001 to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 I0529 14:48:51.450641 1381 tablet_bootstrap.cc:559] Moved log directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236 to recovery directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery W0529 14:48:51.452730 1381 log_util.cc:149] Could not read footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001. Status: Not found: Footer not found. Footer magic doesn't match W0529 14:48:51.453003 1381 log_reader.cc:138] Segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data W0529 14:48:51.455060 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.455304 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.457087 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.457996 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:51.462012 1381 log_util.cc:203] Successfully rebuilt footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 (valid entries through byte offset 8166) W0529 14:48:51.467181 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.467658 1381 log.cc:277] Log is configured to *not* fsync() on all Append() calls W0529 14:48:51.468708 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.471945 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.480373 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.482825 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.484292 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.488883 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.489130 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.490489 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.492297 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.492704 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.496307 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.496825 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.511186 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.515280 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.516047 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.531116 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.533012 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.533519 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.533756 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.544739 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.553776 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.554024 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.557591 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.558804 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.559548 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.578836 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.580953 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.581219 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.596120 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.596846 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.597138 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.598918 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.619148 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.619500 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.622503 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.622886 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.623180 65200 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica aa0d62233d0a498886326226b4827f51 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) W0529 14:48:51.630718 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.634464 1282 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.1:43880 I0529 14:48:51.638422 1282 webserver.cc:108] Starting webserver on 127.245.143.1:51798 I0529 14:48:51.638594 1282 webserver.cc:117] Document root disabled W0529 14:48:51.640712 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.641654 1282 webserver.cc:200] Webserver started. Bound to: http://127.245.143.1:51798/ W0529 14:48:51.641860 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.643584 1282 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/info.pb I0529 14:48:51.645972 1282 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:51.648320 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 1282 I0529 14:48:51.650223 1282 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:51.658785 62863 raft_consensus-itest.cc:805] TS 0 appears to have crashed. Restarting. I0529 14:48:51.659827 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.245.143.0:42480 --local_ip_for_outbound_sockets=127.245.143.0 --tablet_server_web_port=41854 --webserver_interface=127.245.143.0 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --fault_crash_on_leader_request_fraction=0.05 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0529 14:48:51.660151 1609 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:51.660773 1609 heartbeater.cc:312] Registering TS with master... I0529 14:48:51.661280 1609 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:51.664296 64268 ts_manager.cc:71] Re-registered known tablet server { permanent_uuid: "aa0d62233d0a498886326226b4827f51" instance_seqno: 1432936131591352 } with Master kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) W0529 14:48:51.725278 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.747220 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.801575 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.858340 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.886688 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.896302 1617 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:51.735190 1381 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=25} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 18 replicates W0529 14:48:51.930500 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.933846 1381 tablet_bootstrap.cc:945] ReplayState: Previous OpId: 5.43, Committed OpId: 2.25, Pending Replicates: 18, Pending Commits: 0 I0529 14:48:51.933964 1381 tablet_bootstrap.cc:945] Dumping REPLICATES: I0529 14:48:51.934033 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 26 } timestamp: 26 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\330\026\314R_\351\357T\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934103 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 27 } timestamp: 27 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000F\022pu\346\355\277\020\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934172 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 28 } timestamp: 28 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\311\311\2419\256\310\306,\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934243 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 29 } timestamp: 29 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000[\316\375\026\'\304\366p\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934314 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 30 } timestamp: 30 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\3606\006\007Y\333\000\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934386 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 31 } timestamp: 31 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000K\201\323}v\243\315H\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934463 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 32 } timestamp: 32 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\261\370\274L\323Sk\n\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934531 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 33 } timestamp: 33 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0009\245\315^\317q\032\017\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934598 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 34 } timestamp: 34 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\331i\267F\324I\2063\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934667 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 35 } timestamp: 35 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000)L\254:\3275\274\005\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934734 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 36 } timestamp: 36 op_type: NO_OP noop_request { } } I0529 14:48:51.934801 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 37 } timestamp: 37 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000@v?c\247\226l[\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934870 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 38 } timestamp: 38 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\275\2464\330+\327.\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.934936 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 39 } timestamp: 39 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000y.\241.\331!\362W\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.935003 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 40 } timestamp: 40 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0002\271r\007o\205\334|\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.935070 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 41 } timestamp: 41 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\271\027Y5\013\216$l\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.935137 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 42 } timestamp: 42 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\310\324%\021C\237\264J\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:51.935205 1381 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 43 } timestamp: 43 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\241\237\233(\333\027\r\001\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } W0529 14:48:51.937600 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.937835 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.938280 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.934727 1617 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 uuid: "3b7563857bb84965989d41eebdd82d5f" format_stamp: "Formatted at 2015-05-29 21:48:44 on e1205.halxg.cloudera.com" W0529 14:48:51.939281 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.940248 1381 tablet_bootstrap.cc:577] Renamed old recovery dir from: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936131940140 I0529 14:48:51.941098 1381 tablet_bootstrap.cc:585] Removed renamed recovery dir: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936131940140 I0529 14:48:51.941365 1381 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap complete. I0529 14:48:51.941648 1381 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 49ms of latency in Log::Sync() W0529 14:48:51.953464 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.964735 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.967299 1277 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 7 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:51.967962 1277 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 7 FOLLOWER]: Advancing to term 8 I0529 14:48:51.969769 1277 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Snoozing failure detection for an additional: 0.310s I0529 14:48:51.969971 1277 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:51.983080 1277 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 8 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:51.983556 1277 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 8 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f W0529 14:48:51.989436 804 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 3b7563857bb84965989d41eebdd82d5f: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.991314 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:51.993609 806 leader_election.cc:247] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 8 election: Tablet error from VoteRequest() call to peer aa0d62233d0a498886326226b4827f51: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0529 14:48:51.993885 806 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. I0529 14:48:51.995193 1381 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.564s user 0.220s sys 0.077s W0529 14:48:51.995225 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:51.995791 1645 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Snoozing failure detection for an additional: 0.393s I0529 14:48:51.996073 1645 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: None given I0529 14:48:52.002780 1381 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 FOLLOWER]: Replica starting. Triggering 18 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:52.014497 1617 ts_tablet_manager.cc:261] Loading tablet 05b4fe7d1e414fb79249c14e6ae31236 W0529 14:48:52.016827 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.017066 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.022080 1617 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:52.027132 1381 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: FOLLOWER Watermarks: {Received: term: 5 index: 43 Committed: term: 2 index: 25} I0529 14:48:52.027942 1381 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 5.43, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER W0529 14:48:52.029780 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.031754 1381 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.036s user 0.026s sys 0.009s W0529 14:48:52.053205 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.062827 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.110573 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.112037 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.118608 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.145910 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.146177 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.158074 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.183718 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.187602 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.210800 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.222013 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.258678 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.259248 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.259982 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.262295 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.262665 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.278282 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.310323 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.333044 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.334789 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.349649 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.352828 1646 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:52.353307 1646 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 5 FOLLOWER]: Advancing to term 6 I0529 14:48:52.356143 1646 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 6 FOLLOWER]: Snoozing failure detection for an additional: 0.304s I0529 14:48:52.356613 1646 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 6 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } W0529 14:48:52.359462 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.360887 65200 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.362483 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.364259 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.371883 65198 meta_cache.cc:498] GetTableLocations(test-workload, ) failed: Timed out: Call timed out W0529 14:48:52.376067 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.376451 65200 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.377786 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.378263 65200 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.379636 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.383306 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.406924 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.407943 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out I0529 14:48:52.415632 1646 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 6 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:52.416173 1646 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 6 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f W0529 14:48:52.420737 1375 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 3b7563857bb84965989d41eebdd82d5f: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.423102 1034 raft_consensus.cc:1190] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 for earlier term 6. Current term is 8. W0529 14:48:52.425246 1375 leader_election.cc:299] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 6 election: Vote denied by peer 1f8655f18f7e4c6691f4c12ebb22fd06 with higher term. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 for earlier term 6. Current term is 8. I0529 14:48:52.425380 1375 leader_election.cc:302] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term I0529 14:48:52.427098 1655 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 6 FOLLOWER]: Snoozing failure detection for an additional: 0.431s I0529 14:48:52.427376 1655 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 1f8655f18f7e4c6691f4c12ebb22fd06 with higher term. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 for earlier term 6. Current term is 8. W0529 14:48:52.432950 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.434221 1653 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:52.435123 1617 tablet_server_main.cc:41] Starting tablet server... W0529 14:48:52.436663 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.441023 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.442797 1653 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. W0529 14:48:52.449054 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.449367 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.452147 1653 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.004s user 0.003s sys 0.001s I0529 14:48:52.453814 1653 tablet_bootstrap.cc:549] Will attempt to recover log segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236/wal-000000001 to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 I0529 14:48:52.454025 1653 tablet_bootstrap.cc:559] Moved log directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236 to recovery directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery W0529 14:48:52.457787 65200 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.459813 65200 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:52.466114 1653 log_util.cc:149] Could not read footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001. Status: Not found: Footer not found. Footer magic doesn't match W0529 14:48:52.466521 1653 log_reader.cc:138] Segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data W0529 14:48:52.482244 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.482712 1653 log_util.cc:203] Successfully rebuilt footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 (valid entries through byte offset 7974) I0529 14:48:52.491192 1653 log.cc:277] Log is configured to *not* fsync() on all Append() calls W0529 14:48:52.493060 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.493312 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.494799 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.500669 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.514994 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.519951 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.541517 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.554893 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.557802 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.565294 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.567034 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.579257 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.592485 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.616250 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.640651 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.657749 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.672540 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) W0529 14:48:52.673877 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 3b7563857bb84965989d41eebdd82d5f has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:52.683570 1617 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.0:42480 I0529 14:48:52.686353 1617 webserver.cc:108] Starting webserver on 127.245.143.0:41854 I0529 14:48:52.686486 1617 webserver.cc:117] Document root disabled I0529 14:48:52.690103 1617 webserver.cc:200] Webserver started. Bound to: http://127.245.143.0:41854/ I0529 14:48:52.692227 1617 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/info.pb I0529 14:48:52.694795 1617 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:52.698745 1617 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:52.702101 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 1617 I0529 14:48:52.711707 2000 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:52.712347 2000 heartbeater.cc:312] Registering TS with master... I0529 14:48:52.712887 2000 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:52.716243 64264 ts_manager.cc:71] Re-registered known tablet server { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" instance_seqno: 1432936132643029 } with Master I0529 14:48:52.733907 1277 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:52.734347 1277 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 8 FOLLOWER]: Advancing to term 9 I0529 14:48:52.736654 1277 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 FOLLOWER]: Snoozing failure detection for an additional: 0.547s I0529 14:48:52.736876 1277 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:52.750165 1277 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 9 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:52.750602 1277 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 9 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f I0529 14:48:52.752622 1510 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 6 FOLLOWER]: Advancing to term 9 I0529 14:48:52.754458 1510 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Snoozing failure detection for an additional: 0.418s W0529 14:48:52.758522 804 leader_election.cc:247] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 9 election: Tablet error from VoteRequest() call to peer 3b7563857bb84965989d41eebdd82d5f: Illegal state: Tablet not RUNNING: BOOTSTRAPPING I0529 14:48:52.764618 1510 raft_consensus.cc:1261] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f8655f18f7e4c6691f4c12ebb22fd06 in term 9. I0529 14:48:52.765792 806 leader_election.cc:313] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 9 election: Vote granted by peer aa0d62233d0a498886326226b4827f51 I0529 14:48:52.766089 806 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. I0529 14:48:52.768298 2011 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 FOLLOWER]: Snoozing failure detection for an additional: 0.478s I0529 14:48:52.768568 2011 raft_consensus.cc:1396] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 FOLLOWER]: Leader election won for term 9 I0529 14:48:52.770395 2011 raft_consensus.cc:323] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 LEADER]: Becoming Leader. State: Replica: 1f8655f18f7e4c6691f4c12ebb22fd06, State: 1, Role: LEADER Watermarks: {Received: term: 6 index: 41 Committed: term: 2 index: 25} I0529 14:48:52.771200 2011 consensus_queue.cc:109] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: queue going to LEADER mode. State: All replicated op: 6.41, Majority replicated op: 2.25, Committed index: 2.25, Last appended: 6.41, Current term: 9, Majority size: 2, State: 1, Mode: LEADER I0529 14:48:52.782161 993 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 11ms of latency in Log::Sync() I0529 14:48:52.786738 64260 catalog_manager.cc:1393] Tablet: 05b4fe7d1e414fb79249c14e6ae31236 reported consensus state change. New consensus state: current_term: 9 leader_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" quorum { opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } } W0529 14:48:52.797195 2015 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 25, Current term is: 9 I0529 14:48:52.857365 993 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 5ms of latency in Log::Sync() W0529 14:48:52.857735 2012 fault_injection.cc:19] FAULT INJECTION ENABLED! W0529 14:48:52.857892 2012 fault_injection.cc:20] THIS SERVER MAY CRASH! F0529 14:48:52.858023 2012 fault_injection.cc:36] Injected fault: FLAGS_fault_crash_on_leader_request_fraction *** Check failure stack trace: *** I0529 14:48:52.860800 1511 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Refusing update from remote peer 1f8655f18f7e4c6691f4c12ebb22fd06: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 9 index: 43. (term mismatch) I0529 14:48:52.860975 1511 raft_consensus_state.cc:340] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Aborting all transactions after (but not including): 42. Current State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: FOLLOWER Watermarks: {Received: term: 5 index: 43 Committed: term: 2 index: 25} I0529 14:48:52.861294 1511 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.43 @ 0x7fb042330bca google::LogMessage::Fail() W0529 14:48:52.863839 2015 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 25, Current term is: 9 I0529 14:48:52.864632 2011 consensus_queue.cc:420] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: Connected to new peer: Peer: aa0d62233d0a498886326226b4827f51, Is new: false, Last received: 5.42, Next index: 43, Last known committed idx: 25, Last exchange result: ERROR @ 0x7fb042330496 google::LogMessage::SendToLog() I0529 14:48:52.868437 1514 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Refusing update from remote peer 1f8655f18f7e4c6691f4c12ebb22fd06: Log matching property violated. Preceding OpId in replica: term: 5 index: 42. Preceding OpId from leader: term: 9 index: 42. (term mismatch) I0529 14:48:52.868610 1514 raft_consensus_state.cc:340] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Aborting all transactions after (but not including): 41. Current State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: FOLLOWER Watermarks: {Received: term: 5 index: 42 Committed: term: 2 index: 25} I0529 14:48:52.868852 1514 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.42 @ 0x7fb0423308ee google::LogMessage::Flush() I0529 14:48:52.871089 2011 consensus_queue.cc:422] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: Got LMP mismatch error from peer: Peer: aa0d62233d0a498886326226b4827f51, Is new: false, Last received: 5.41, Next index: 42, Last known committed idx: 25, Last exchange result: ERROR @ 0x7fb04233408c google::LogMessageFatal::~LogMessageFatal() I0529 14:48:52.874321 1516 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Refusing update from remote peer 1f8655f18f7e4c6691f4c12ebb22fd06: Log matching property violated. Preceding OpId in replica: term: 5 index: 41. Preceding OpId from leader: term: 6 index: 41. (term mismatch) I0529 14:48:52.874610 1516 raft_consensus_state.cc:340] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Aborting all transactions after (but not including): 40. Current State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: FOLLOWER Watermarks: {Received: term: 5 index: 41 Committed: term: 2 index: 25} I0529 14:48:52.875005 1516 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.41 I0529 14:48:52.878356 2011 consensus_queue.cc:422] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: Got LMP mismatch error from peer: Peer: aa0d62233d0a498886326226b4827f51, Is new: false, Last received: 5.40, Next index: 41, Last known committed idx: 25, Last exchange result: ERROR @ 0x7fb03b22a45f kudu::fault_injection::DoMaybeFault() I0529 14:48:52.881113 1519 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Refusing update from remote peer 1f8655f18f7e4c6691f4c12ebb22fd06: Log matching property violated. Preceding OpId in replica: term: 5 index: 40. Preceding OpId from leader: term: 6 index: 40. (term mismatch) I0529 14:48:52.881294 1519 raft_consensus_state.cc:340] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Aborting all transactions after (but not including): 39. Current State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: FOLLOWER Watermarks: {Received: term: 5 index: 40 Committed: term: 2 index: 25} I0529 14:48:52.881575 1519 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.40 @ 0x7fb03fe5189c kudu::fault_injection::MaybeFault() I0529 14:48:52.883589 2011 consensus_queue.cc:422] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: Got LMP mismatch error from peer: Peer: aa0d62233d0a498886326226b4827f51, Is new: false, Last received: 5.39, Next index: 40, Last known committed idx: 25, Last exchange result: ERROR F0529 14:48:52.884289 2011 fault_injection.cc:36] Injected fault: FLAGS_fault_crash_on_leader_request_fraction *** Check failure stack trace: *** @ 0x7fb03fe4e9da kudu::consensus::Peer::SendNextRequest() @ 0x7fb042330bca google::LogMessage::Fail() @ 0x7fb03fe53f38 kudu::internal::RunnableAdapter<>::Run() @ 0x7fb042330496 google::LogMessage::SendToLog() @ 0x7fb03fe53e7f kudu::internal::InvokeHelper<>::MakeItSo() @ 0x7fb0423308ee google::LogMessage::Flush() @ 0x7fb03fe53db4 kudu::internal::Invoker<>::Run() @ 0x7fb04233408c google::LogMessageFatal::~LogMessageFatal() @ 0x7fb03fe717be kudu::Callback<>::Run() @ 0x7fb03b22a45f kudu::fault_injection::DoMaybeFault() @ 0x7fb03fe724fe boost::_mfi::cmf0<>::operator()() @ 0x7fb03fe5189c kudu::fault_injection::MaybeFault() @ 0x7fb03fe7246f boost::_bi::list1<>::operator()<>() @ 0x7fb03fe4e9da kudu::consensus::Peer::SendNextRequest() @ 0x7fb03fe72424 boost::_bi::bind_t<>::operator()() @ 0x7fb03fe4f5a9 kudu::consensus::Peer::DoProcessResponse() @ 0x7fb03fe721b2 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7fb03fe5317e kudu::internal::RunnableAdapter<>::Run() @ 0x7fb03ef843e2 boost::function0<>::operator()() @ 0x7fb03fe530ec kudu::internal::InvokeHelper<>::MakeItSo() @ 0x7fb03b2a771e kudu::FunctionRunnable::Run() @ 0x7fb03fe5301b kudu::internal::Invoker<>::Run() @ 0x7fb03b2a4551 kudu::ThreadPool::DispatchThread() @ 0x7fb03fe717be kudu::Callback<>::Run() I0529 14:48:52.761584 1653 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=25} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 18 replicates @ 0x7fb03b2a5ca6 boost::_mfi::mf1<>::operator()() @ 0x7fb03fe724fe boost::_mfi::cmf0<>::operator()() @ 0x7fb03b2a5c0c boost::_bi::list2<>::operator()<>() @ 0x7fb03fe7246f boost::_bi::list1<>::operator()<>() @ 0x7fb03b2a5b94 boost::_bi::bind_t<>::operator()() @ 0x7fb03fe72424 boost::_bi::bind_t<>::operator()() @ 0x7fb03b2a5992 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7fb03fe721b2 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x7fb03ef843e2 boost::function0<>::operator()() @ 0x7fb03ef843e2 boost::function0<>::operator()() @ 0x7fb03b2a771e kudu::FunctionRunnable::Run() I0529 14:48:52.954324 993 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 88ms of latency in Log::Sync() @ 0x7fb03b299d28 kudu::Thread::SuperviseThread() I0529 14:48:52.956153 1653 tablet_bootstrap.cc:945] ReplayState: Previous OpId: 6.43, Committed OpId: 2.25, Pending Replicates: 18, Pending Commits: 0 I0529 14:48:52.956732 1653 tablet_bootstrap.cc:945] Dumping REPLICATES: I0529 14:48:52.956809 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 26 } timestamp: 26 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\330\026\314R_\351\357T\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.956895 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 27 } timestamp: 27 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000F\022pu\346\355\277\020\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.956981 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 28 } timestamp: 28 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\311\311\2419\256\310\306,\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957067 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 29 } timestamp: 29 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000[\316\375\026\'\304\366p\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957149 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 30 } timestamp: 30 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\3606\006\007Y\333\000\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957217 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 31 } timestamp: 31 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000K\201\323}v\243\315H\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957298 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 32 } timestamp: 32 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\261\370\274L\323Sk\n\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957366 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 33 } timestamp: 33 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0009\245\315^\317q\032\017\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957453 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 34 } timestamp: 34 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\331i\267F\324I\2063\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957546 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 35 } timestamp: 35 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000)L\254:\3275\274\005\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957648 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 36 } timestamp: 36 op_type: NO_OP noop_request { } } I0529 14:48:52.957762 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 37 } timestamp: 37 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000@v?c\247\226l[\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957877 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 38 } timestamp: 38 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\275\2464\330+\327.\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.957993 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 39 } timestamp: 39 op_type: NO_OP noop_request { } } I0529 14:48:52.958104 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 40 } timestamp: 40 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000a\026\310X\320g58\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.958230 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 41 } timestamp: 41 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\375FL\014Y\222\310c\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.958338 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 42 } timestamp: 42 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0002\271r\007o\205\334|\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:52.958415 1653 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 43 } timestamp: 43 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\271\027Y5\013\216$l\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } @ 0x7fb03b2a4551 kudu::ThreadPool::DispatchThread() @ 0x7fb042b37ea2 __tsan::cur_thread_placeholder @ 0x7fb03b2a5ca6 boost::_mfi::mf1<>::operator()() I0529 14:48:52.963737 1653 tablet_bootstrap.cc:577] Renamed old recovery dir from: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936132963637 I0529 14:48:52.964594 1653 tablet_bootstrap.cc:585] Removed renamed recovery dir: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936132963637 @ 0x7fb03c89a851 (unknown) I0529 14:48:52.964874 1653 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap complete. I0529 14:48:52.965219 1653 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 15ms of latency in Log::Sync() @ 0x7fb03b2a5c0c boost::_bi::list2<>::operator()<>() @ 0x7fb03944c94d (unknown) @ 0x7fb03b2a5b94 boost::_bi::bind_t<>::operator()() @ 0x7fb03b2a5992 boost::detail::function::void_function_obj_invoker0<>::invoke() @ (nil) (unknown) I0529 14:48:52.983530 1653 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.549s user 0.234s sys 0.088s I0529 14:48:52.991260 1653 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 FOLLOWER]: Replica starting. Triggering 18 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:53.009433 1653 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7563857bb84965989d41eebdd82d5f, State: 1, Role: FOLLOWER Watermarks: {Received: term: 6 index: 43 Committed: term: 2 index: 25} I0529 14:48:53.009943 1653 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 6.43, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:53.013823 1653 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.030s user 0.025s sys 0.003s W0529 14:48:53.037390 65198 connection.cc:379] client Connection to 127.245.143.2:53754 recv error: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.037827 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.038127 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.038337 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.038571 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.038789 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.038995 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.039233 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.039449 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.039659 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.039860 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: recv error: Connection reset by peer (error 104) W0529 14:48:53.068619 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.089884 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.090206 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.090761 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.118012 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.118320 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.119556 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.127450 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.127802 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.128090 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.129343 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.144412 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.154407 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.156605 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.169651 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.183295 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.198038 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.211549 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.222777 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.237190 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.253711 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.277868 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.278169 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.283599 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.307838 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.308156 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.316819 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:53.330467 2524 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:53.331491 2524 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 6 FOLLOWER]: Advancing to term 7 I0529 14:48:53.334992 2524 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 7 FOLLOWER]: Snoozing failure detection for an additional: 0.388s I0529 14:48:53.335377 2524 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 7 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } W0529 14:48:53.337043 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.337671 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:53.349329 2524 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 7 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:53.349907 2524 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 7 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 W0529 14:48:53.354925 1647 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:53.358883 1521 raft_consensus.cc:1190] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for earlier term 7. Current term is 9. W0529 14:48:53.360350 1650 leader_election.cc:299] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 7 election: Vote denied by peer aa0d62233d0a498886326226b4827f51 with higher term. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for earlier term 7. Current term is 9. I0529 14:48:53.360522 1650 leader_election.cc:302] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term I0529 14:48:53.361531 2526 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 7 FOLLOWER]: Snoozing failure detection for an additional: 0.413s I0529 14:48:53.361850 2526 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer aa0d62233d0a498886326226b4827f51 with higher term. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for earlier term 7. Current term is 9. W0529 14:48:53.367777 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.374284 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.374891 65200 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.374917 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.386791 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.388087 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.400013 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.404072 65198 meta_cache.cc:164] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Replica 1f8655f18f7e4c6691f4c12ebb22fd06 has failed: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.415346 65200 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.422638 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.426151 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.475983 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out W0529 14:48:53.483175 65198 batcher.cc:429] Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) failed: Timed out: Write(05b4fe7d1e414fb79249c14e6ae31236, 1 ops) timed out I0529 14:48:53.489720 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server with pid 1617 I0529 14:48:53.501417 1646 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:53.501888 1646 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 9 FOLLOWER]: Advancing to term 10 I0529 14:48:53.504374 1646 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 10 FOLLOWER]: Snoozing failure detection for an additional: 0.605s I0529 14:48:53.504593 1646 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 10 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:53.514453 1646 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 10 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:53.514997 1646 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 10 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f W0529 14:48:53.518769 1375 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:53.538645 1375 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 3b7563857bb84965989d41eebdd82d5f: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: BlockingRecv error: recv error: Connection reset by peer (error 104) I0529 14:48:53.539389 1375 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. I0529 14:48:53.541801 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.245.143.0:42480 --local_ip_for_outbound_sockets=127.245.143.0 --tablet_server_web_port=41854 --webserver_interface=127.245.143.0 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0529 14:48:53.542188 2578 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 10 FOLLOWER]: Snoozing failure detection for an additional: 0.343s I0529 14:48:53.542539 2578 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: None given kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:53.761255 2579 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:53.798391 2579 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0 uuid: "3b7563857bb84965989d41eebdd82d5f" format_stamp: "Formatted at 2015-05-29 21:48:44 on e1205.halxg.cloudera.com" I0529 14:48:53.873785 2579 ts_tablet_manager.cc:261] Loading tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:53.884657 2579 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:54.186652 1646 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 10 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:54.187137 1646 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 10 FOLLOWER]: Advancing to term 11 I0529 14:48:54.189508 1646 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 11 FOLLOWER]: Snoozing failure detection for an additional: 0.570s I0529 14:48:54.189754 1646 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 11 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:54.197191 1646 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 11 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:54.197763 1646 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 11 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f W0529 14:48:54.201267 1375 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:54.201935 1375 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 3b7563857bb84965989d41eebdd82d5f: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111) I0529 14:48:54.202247 1375 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. I0529 14:48:54.204308 2822 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 11 FOLLOWER]: Snoozing failure detection for an additional: 0.388s I0529 14:48:54.204778 2822 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: None given I0529 14:48:54.281113 2701 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:54.282403 2579 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:54.288846 2701 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. I0529 14:48:54.296916 2701 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.003s user 0.002s sys 0.001s I0529 14:48:54.298143 2701 tablet_bootstrap.cc:549] Will attempt to recover log segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236/wal-000000001 to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 I0529 14:48:54.298343 2701 tablet_bootstrap.cc:559] Moved log directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236 to recovery directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery W0529 14:48:54.300743 2701 log_util.cc:149] Could not read footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001. Status: Not found: Footer not found. Footer magic doesn't match W0529 14:48:54.301009 2701 log_reader.cc:138] Segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data I0529 14:48:54.310547 2701 log_util.cc:203] Successfully rebuilt footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 (valid entries through byte offset 8358) I0529 14:48:54.315258 2701 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:54.504833 2579 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.0:42480 I0529 14:48:54.507365 2579 webserver.cc:108] Starting webserver on 127.245.143.0:41854 I0529 14:48:54.507478 2579 webserver.cc:117] Document root disabled I0529 14:48:54.510414 2579 webserver.cc:200] Webserver started. Bound to: http://127.245.143.0:41854/ I0529 14:48:54.512378 2579 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/info.pb I0529 14:48:54.514113 2579 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:54.514374 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 2579 I0529 14:48:54.514883 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server with pid 1282 I0529 14:48:54.518687 2579 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:54.531299 2975 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:54.531905 2975 heartbeater.cc:312] Registering TS with master... I0529 14:48:54.532444 2975 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:54.535305 64256 ts_manager.cc:71] Re-registered known tablet server { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" instance_seqno: 1432936134461724 } with Master I0529 14:48:54.548270 2701 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=25} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 18 replicates I0529 14:48:54.577008 2701 tablet_bootstrap.cc:945] ReplayState: Previous OpId: 6.43, Committed OpId: 2.25, Pending Replicates: 18, Pending Commits: 0 I0529 14:48:54.577126 2701 tablet_bootstrap.cc:945] Dumping REPLICATES: I0529 14:48:54.577198 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 26 } timestamp: 26 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\330\026\314R_\351\357T\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.577283 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 27 } timestamp: 27 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000F\022pu\346\355\277\020\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.577373 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 28 } timestamp: 28 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\311\311\2419\256\310\306,\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.577674 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 29 } timestamp: 29 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000[\316\375\026\'\304\366p\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.577749 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 30 } timestamp: 30 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\3606\006\007Y\333\000\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.577821 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 31 } timestamp: 31 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000K\201\323}v\243\315H\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.577889 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 32 } timestamp: 32 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\261\370\274L\323Sk\n\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.577957 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 33 } timestamp: 33 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0009\245\315^\317q\032\017\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.578024 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 34 } timestamp: 34 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\331i\267F\324I\2063\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.578094 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 35 } timestamp: 35 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000)L\254:\3275\274\005\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.578163 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 36 } timestamp: 36 op_type: NO_OP noop_request { } } I0529 14:48:54.578232 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 37 } timestamp: 37 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000@v?c\247\226l[\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.578299 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 38 } timestamp: 38 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\275\2464\330+\327.\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.578367 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 39 } timestamp: 39 op_type: NO_OP noop_request { } } I0529 14:48:54.578462 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 40 } timestamp: 40 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000a\026\310X\320g58\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.578531 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 41 } timestamp: 41 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\375FL\014Y\222\310c\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.578605 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 42 } timestamp: 42 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0002\271r\007o\205\334|\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.578680 2701 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 43 } timestamp: 43 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\271\027Y5\013\216$l\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:54.582867 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.245.143.1:43880 --local_ip_for_outbound_sockets=127.245.143.1 --tablet_server_web_port=51798 --webserver_interface=127.245.143.1 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0529 14:48:54.583091 2701 tablet_bootstrap.cc:577] Renamed old recovery dir from: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936134582992 I0529 14:48:54.583932 2701 tablet_bootstrap.cc:585] Removed renamed recovery dir: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-0/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936134582992 I0529 14:48:54.584210 2701 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap complete. I0529 14:48:54.584525 2701 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 124ms of latency in Log::Sync() kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:54.712380 2701 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.431s user 0.220s sys 0.072s I0529 14:48:54.719970 2701 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 7 FOLLOWER]: Replica starting. Triggering 18 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:54.737355 2701 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7563857bb84965989d41eebdd82d5f, State: 1, Role: FOLLOWER Watermarks: {Received: term: 6 index: 43 Committed: term: 2 index: 25} I0529 14:48:54.737905 2701 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 6.43, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:54.742704 2701 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.030s user 0.023s sys 0.006s I0529 14:48:54.806660 2988 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:54.843405 2988 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1 uuid: "aa0d62233d0a498886326226b4827f51" format_stamp: "Formatted at 2015-05-29 21:48:44 on e1205.halxg.cloudera.com" I0529 14:48:54.911736 2988 ts_tablet_manager.cc:261] Loading tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:54.918074 2988 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:55.074347 2992 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 7 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:55.075124 2992 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 7 FOLLOWER]: Advancing to term 8 I0529 14:48:55.078284 2992 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 8 FOLLOWER]: Snoozing failure detection for an additional: 0.433s I0529 14:48:55.078857 2992 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 8 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:55.086675 2992 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 8 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:55.087175 2992 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 8 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 W0529 14:48:55.092072 2684 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) W0529 14:48:55.092766 2689 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer aa0d62233d0a498886326226b4827f51: Network error: Client connection negotiation failed: client Connection to 127.245.143.1:43880: connect: Connection refused (error 111) I0529 14:48:55.093322 2689 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. I0529 14:48:55.094631 2997 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 8 FOLLOWER]: Snoozing failure detection for an additional: 0.302s I0529 14:48:55.095047 2997 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: None given I0529 14:48:55.339648 3120 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:55.341373 2988 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:55.348021 3120 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. I0529 14:48:55.355108 3120 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.002s user 0.002s sys 0.000s I0529 14:48:55.356855 3120 tablet_bootstrap.cc:549] Will attempt to recover log segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236/wal-000000001 to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 I0529 14:48:55.357055 3120 tablet_bootstrap.cc:559] Moved log directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236 to recovery directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery W0529 14:48:55.359410 3120 log_util.cc:149] Could not read footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001. Status: Not found: Footer not found. Footer magic doesn't match W0529 14:48:55.359679 3120 log_reader.cc:138] Segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data I0529 14:48:55.369042 3120 log_util.cc:203] Successfully rebuilt footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 (valid entries through byte offset 8502) I0529 14:48:55.373680 3120 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:55.552248 2988 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.1:43880 I0529 14:48:55.555006 2988 webserver.cc:108] Starting webserver on 127.245.143.1:51798 I0529 14:48:55.555178 2988 webserver.cc:117] Document root disabled I0529 14:48:55.558289 2988 webserver.cc:200] Webserver started. Bound to: http://127.245.143.1:51798/ I0529 14:48:55.559809 2988 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/info.pb I0529 14:48:55.561530 2988 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:55.563902 2988 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:55.566162 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 2988 I0529 14:48:55.567281 62863 external_mini_cluster.cc:425] Running /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server kudu-tablet_server --tablet_server_wal_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --tablet_server_data_dirs=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.245.143.2:53754 --local_ip_for_outbound_sockets=127.245.143.2 --tablet_server_web_port=34287 --webserver_interface=127.245.143.2 --tablet_server_master_addrs=127.0.0.1:59882 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 --log_inject_latency --log_inject_latency_ms_mean=30 --log_inject_latency_ms_stddev=60 --leader_heartbeat_interval_ms=100 --leader_failure_monitor_check_mean_ms=50 --leader_failure_monitor_check_stddev_ms=25 --log_preallocate_segments=false --server_dump_info_path=/tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0529 14:48:55.575361 3281 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:55.576170 3281 heartbeater.cc:312] Registering TS with master... I0529 14:48:55.576669 3281 heartbeater.cc:318] Sending a full tablet report to master... I0529 14:48:55.579196 64266 ts_manager.cc:71] Re-registered known tablet server { permanent_uuid: "aa0d62233d0a498886326226b4827f51" instance_seqno: 1432936135513221 } with Master kudu-tablet_server: /usr/lib64/libssl.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) kudu-tablet_server: /usr/lib64/libcrypto.so.10: no version information available (required by /home/todd/run_tha_testHSjhNC/build/fastdebug/libserver_process.so) I0529 14:48:55.612797 3120 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=25} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 18 replicates I0529 14:48:55.643335 3120 tablet_bootstrap.cc:945] ReplayState: Previous OpId: 5.43, Committed OpId: 2.25, Pending Replicates: 18, Pending Commits: 0 I0529 14:48:55.643448 3120 tablet_bootstrap.cc:945] Dumping REPLICATES: I0529 14:48:55.643527 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 26 } timestamp: 26 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\330\026\314R_\351\357T\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.643609 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 27 } timestamp: 27 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000F\022pu\346\355\277\020\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.643692 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 28 } timestamp: 28 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\311\311\2419\256\310\306,\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.643789 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 29 } timestamp: 29 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000[\316\375\026\'\304\366p\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.643860 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 30 } timestamp: 30 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\3606\006\007Y\333\000\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.643929 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 31 } timestamp: 31 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000K\201\323}v\243\315H\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644002 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 32 } timestamp: 32 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\261\370\274L\323Sk\n\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644073 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 33 } timestamp: 33 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0009\245\315^\317q\032\017\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644143 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 34 } timestamp: 34 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\331i\267F\324I\2063\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644212 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 35 } timestamp: 35 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000)L\254:\3275\274\005\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644294 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 36 } timestamp: 36 op_type: NO_OP noop_request { } } I0529 14:48:55.644364 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 37 } timestamp: 37 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000@v?c\247\226l[\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644435 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 38 } timestamp: 38 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\275\2464\330+\327.\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644510 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 39 } timestamp: 39 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000y.\241.\331!\362W\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644587 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 40 } timestamp: 40 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0002\271r\007o\205\334|\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644661 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 41 } timestamp: 41 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\271\027Y5\013\216$l\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644774 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 42 } timestamp: 42 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\310\324%\021C\237\264J\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.644860 3120 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 43 } timestamp: 43 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\241\237\233(\333\027\r\001\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:55.650315 3120 tablet_bootstrap.cc:577] Renamed old recovery dir from: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936135650164 I0529 14:48:55.651336 3120 tablet_bootstrap.cc:585] Removed renamed recovery dir: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-1/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936135650164 I0529 14:48:55.651680 3120 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap complete. I0529 14:48:55.654471 3120 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.314s user 0.215s sys 0.083s I0529 14:48:55.661548 3120 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 11 FOLLOWER]: Replica starting. Triggering 18 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:55.680131 3120 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: FOLLOWER Watermarks: {Received: term: 5 index: 43 Committed: term: 2 index: 25} I0529 14:48:55.680634 3120 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 5.43, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:55.684973 3120 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.030s user 0.018s sys 0.010s I0529 14:48:55.728415 2992 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 8 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:55.728920 2992 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 8 FOLLOWER]: Advancing to term 9 I0529 14:48:55.813527 3287 tablet_server_main.cc:38] Initializing tablet server... I0529 14:48:55.851444 3287 fs_manager.cc:198] Opened local filesystem: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2 uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" format_stamp: "Formatted at 2015-05-29 21:48:45 on e1205.halxg.cloudera.com" I0529 14:48:55.935485 3287 ts_tablet_manager.cc:261] Loading tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:55.943315 3287 ts_tablet_manager.cc:406] Registered tablet 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:56.031150 3302 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 11 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:56.031816 3302 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 11 FOLLOWER]: Advancing to term 12 I0529 14:48:56.035218 3302 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 12 FOLLOWER]: Snoozing failure detection for an additional: 0.632s I0529 14:48:56.036074 3302 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 12 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:56.045040 3302 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 12 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:56.046874 3302 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 12 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f W0529 14:48:56.049824 3107 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:56.229286 2992 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 9 FOLLOWER]: Snoozing failure detection for an additional: 0.509s I0529 14:48:56.229574 2992 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 9 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:56.235091 2992 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 9 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:56.236948 2992 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 9 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 W0529 14:48:56.239110 2684 leader_election.cc:241] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 1f8655f18f7e4c6691f4c12ebb22fd06: Network error: Client connection negotiation failed: client Connection to 127.245.143.2:53754: connect: Connection refused (error 111) I0529 14:48:56.239171 2891 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 9 FOLLOWER]: Advancing to term 12 I0529 14:48:56.243520 2891 raft_consensus.cc:1232] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 for term 12 because replica has last-logged OpId of term: 6 index: 43, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 43. I0529 14:48:56.248392 3107 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 12 election: Vote denied by peer 3b7563857bb84965989d41eebdd82d5f. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0d62233d0a498886326226b4827f51 for term 12 because replica has last-logged OpId of term: 6 index: 43, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 43. I0529 14:48:56.248690 3208 raft_consensus.cc:1190] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for earlier term 9. Current term is 12. I0529 14:48:56.248836 3107 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. I0529 14:48:56.251502 3653 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 12 FOLLOWER]: Snoozing failure detection for an additional: 0.598s I0529 14:48:56.251780 3653 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: None given W0529 14:48:56.253023 2689 leader_election.cc:299] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 9 election: Vote denied by peer aa0d62233d0a498886326226b4827f51 with higher term. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for earlier term 9. Current term is 12. I0529 14:48:56.253283 2689 leader_election.cc:302] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term I0529 14:48:56.256672 3654 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 12 FOLLOWER]: Snoozing failure detection for an additional: 0.327s I0529 14:48:56.256908 3654 raft_consensus.cc:1358] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 12 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer aa0d62233d0a498886326226b4827f51 with higher term. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for earlier term 9. Current term is 12. I0529 14:48:56.350805 3527 ts_tablet_manager.cc:284] Bootstrapping tablet: 05b4fe7d1e414fb79249c14e6ae31236 I0529 14:48:56.351994 3287 tablet_server_main.cc:41] Starting tablet server... I0529 14:48:56.359839 3527 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap starting. I0529 14:48:56.369283 3527 tablet_bootstrap.cc:472] Time spent opening tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.004s user 0.003s sys 0.001s I0529 14:48:56.370822 3527 tablet_bootstrap.cc:549] Will attempt to recover log segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236/wal-000000001 to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 I0529 14:48:56.371070 3527 tablet_bootstrap.cc:559] Moved log directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236 to recovery directory: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery W0529 14:48:56.373832 3527 log_util.cc:149] Could not read footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001. Status: Not found: Footer not found. Footer magic doesn't match W0529 14:48:56.374145 3527 log_reader.cc:138] Segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data I0529 14:48:56.385349 3527 log_util.cc:203] Successfully rebuilt footer for segment: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery/wal-000000001 (valid entries through byte offset 8252) I0529 14:48:56.392696 3527 log.cc:277] Log is configured to *not* fsync() on all Append() calls I0529 14:48:56.578984 3287 rpc_server.cc:110] RPC server started. Bound to: 127.245.143.2:53754 I0529 14:48:56.581804 3287 webserver.cc:108] Starting webserver on 127.245.143.2:34287 I0529 14:48:56.581938 3287 webserver.cc:117] Document root disabled I0529 14:48:56.584765 3287 webserver.cc:200] Webserver started. Bound to: http://127.245.143.2:34287/ I0529 14:48:56.587749 3287 server_base.cc:201] Dumped server information to /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/info.pb I0529 14:48:56.589761 3287 maintenance_manager.cc:114] MaintenanceManager: targeting memory size of 50.302254 GB I0529 14:48:56.592607 3287 tablet_server_main.cc:44] Tablet server successfully started. I0529 14:48:56.597713 62863 external_mini_cluster.cc:460] Started /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server as pid 3287 I0529 14:48:56.610381 3800 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:59882 I0529 14:48:56.611074 3800 heartbeater.cc:312] Registering TS with master... I0529 14:48:56.611609 3800 heartbeater.cc:318] Sending a full tablet report to master... INFO: Connected to the Master I0529 14:48:56.614121 64262 ts_manager.cc:71] Re-registered known tablet server { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" instance_seqno: 1432936136531807 } with Master I0529 14:48:56.650566 3527 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=25} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 19 replicates I0529 14:48:56.678247 3527 tablet_bootstrap.cc:945] ReplayState: Previous OpId: 9.44, Committed OpId: 2.25, Pending Replicates: 19, Pending Commits: 0 I0529 14:48:56.678350 3527 tablet_bootstrap.cc:945] Dumping REPLICATES: I0529 14:48:56.678431 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 26 } timestamp: 26 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\330\026\314R_\351\357T\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.678501 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 27 } timestamp: 27 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000F\022pu\346\355\277\020\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.678594 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 28 } timestamp: 28 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\311\311\2419\256\310\306,\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.678684 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 29 } timestamp: 29 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000[\316\375\026\'\304\366p\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.678764 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 30 } timestamp: 30 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\3606\006\007Y\333\000\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.678836 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 31 } timestamp: 31 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000K\201\323}v\243\315H\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.678922 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 32 } timestamp: 32 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\261\370\274L\323Sk\n\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.678994 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 33 } timestamp: 33 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\0009\245\315^\317q\032\017\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.679066 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 34 } timestamp: 34 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\331i\267F\324I\2063\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.679134 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 2 index: 35 } timestamp: 35 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000)L\254:\3275\274\005\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.679201 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 36 } timestamp: 36 op_type: NO_OP noop_request { } } I0529 14:48:56.679266 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 37 } timestamp: 37 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000@v?c\247\226l[\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.679334 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 5 index: 38 } timestamp: 38 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000Q\275\2464\330+\327.\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.679419 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 39 } timestamp: 39 op_type: NO_OP noop_request { } } I0529 14:48:56.679488 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 40 } timestamp: 40 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000a\026\310X\320g58\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.679555 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 6 index: 41 } timestamp: 41 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\375FL\014Y\222\310c\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.679662 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 9 index: 42 } timestamp: 42 op_type: NO_OP noop_request { } } I0529 14:48:56.679769 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 9 index: 43 } timestamp: 43 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\014\023\323\"\260\212\207O\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.679883 3527 tablet_bootstrap.cc:945] type: REPLICATE replicate { id { term: 9 index: 44 } timestamp: 44 op_type: WRITE_OP write_request { tablet_id: "05b4fe7d1e414fb79249c14e6ae31236" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } row_operations { rows: "\001\007\000\'d\3775#X\010\022\000\000\000\000\000\000\000\000\013\000\000\000\000\000\000\000" indirect_data: "hello world" } } } I0529 14:48:56.686276 3527 tablet_bootstrap.cc:577] Renamed old recovery dir from: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery to: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936136686141 I0529 14:48:56.687187 3527 tablet_bootstrap.cc:585] Removed renamed recovery dir: /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863/raft_consensus-itest-cluster/ts-2/wals/05b4fe7d1e414fb79249c14e6ae31236.recovery-1432936136686141 I0529 14:48:56.687702 3527 tablet_bootstrap.cc:311] Tablet 05b4fe7d1e414fb79249c14e6ae31236: Bootstrap complete. I0529 14:48:56.690524 3527 ts_tablet_manager.cc:289] Time spent bootstrapping tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.339s user 0.253s sys 0.070s INFO: Connected to all 3 Tablet Servers WARNING: Tablet 05b4fe7d1e414fb79249c14e6ae31236 is missing 1 replicas WARNING: Tablet 05b4fe7d1e414fb79249c14e6ae31236 doesn't have a leader WARNING: Table test-workload has 1 bad tablets WARNING: 1 out of 1 tables are not in a healthy state I0529 14:48:56.698971 3527 raft_consensus.cc:182] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 FOLLOWER]: Replica starting. Triggering 19 pending transactions. Active quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:56.701522 62863 cluster_verifier.cc:53] Check not successful yet, sleeping and retrying: Corruption: 1 tables are bad I0529 14:48:56.722764 3527 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8655f18f7e4c6691f4c12ebb22fd06, State: 1, Role: FOLLOWER Watermarks: {Received: term: 9 index: 44 Committed: term: 2 index: 25} I0529 14:48:56.723273 3527 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 9.44, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:56.728672 3527 ts_tablet_manager.cc:310] Time spent starting tablet 05b4fe7d1e414fb79249c14e6ae31236: real 0.038s user 0.025s sys 0.010s INFO: Connected to the Master I0529 14:48:56.921991 2992 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 12 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:56.922680 2992 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 12 FOLLOWER]: Advancing to term 13 I0529 14:48:56.926897 2992 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 FOLLOWER]: Snoozing failure detection for an additional: 0.702s I0529 14:48:56.927129 2992 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } INFO: Connected to all 3 Tablet Servers WARNING: Tablet 05b4fe7d1e414fb79249c14e6ae31236 doesn't have a leader WARNING: Table test-workload has 1 bad tablets WARNING: 1 out of 1 tables are not in a healthy state I0529 14:48:56.934494 2992 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 13 election: Requesting vote from peer 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:56.935173 2992 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 13 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:56.936532 62863 cluster_verifier.cc:53] Check not successful yet, sleeping and retrying: Corruption: 1 tables are bad I0529 14:48:56.937836 3209 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 12 FOLLOWER]: Advancing to term 13 I0529 14:48:56.940762 3209 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 13 FOLLOWER]: Snoozing failure detection for an additional: 0.728s I0529 14:48:56.941231 3722 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 9 FOLLOWER]: Advancing to term 13 I0529 14:48:56.943737 3722 raft_consensus.cc:1232] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for term 13 because replica has last-logged OpId of term: 9 index: 44, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 43. I0529 14:48:56.945034 2684 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 13 election: Vote denied by peer 1f8655f18f7e4c6691f4c12ebb22fd06. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b7563857bb84965989d41eebdd82d5f for term 13 because replica has last-logged OpId of term: 9 index: 44, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 43. I0529 14:48:56.951280 3209 raft_consensus.cc:1261] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b7563857bb84965989d41eebdd82d5f in term 13. I0529 14:48:56.952750 2689 leader_election.cc:313] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 13 election: Vote granted by peer aa0d62233d0a498886326226b4827f51 I0529 14:48:56.953131 2689 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. I0529 14:48:56.956639 3971 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 FOLLOWER]: Snoozing failure detection for an additional: 0.462s I0529 14:48:56.957123 3971 raft_consensus.cc:1396] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 FOLLOWER]: Leader election won for term 13 I0529 14:48:56.962908 3971 raft_consensus.cc:323] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 LEADER]: Becoming Leader. State: Replica: 3b7563857bb84965989d41eebdd82d5f, State: 1, Role: LEADER Watermarks: {Received: term: 6 index: 43 Committed: term: 2 index: 25} I0529 14:48:56.964834 3971 consensus_queue.cc:109] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [LEADER]: queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 2.25, Committed index: 2.25, Last appended: 6.43, Current term: 13, Majority size: 2, State: 1, Mode: LEADER I0529 14:48:56.976560 2857 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 7ms of latency in Log::Sync() I0529 14:48:56.980259 64269 catalog_manager.cc:1393] Tablet: 05b4fe7d1e414fb79249c14e6ae31236 reported consensus state change. New consensus state: current_term: 13 leader_uuid: "3b7563857bb84965989d41eebdd82d5f" quorum { opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } } W0529 14:48:56.989826 4004 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 25, Current term is: 13 I0529 14:48:57.046962 3834 raft_consensus.cc:310] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 13 FOLLOWER]: Failure of peer election-timer detected. Triggering leader election I0529 14:48:57.047778 3834 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 13 FOLLOWER]: Advancing to term 14 I0529 14:48:57.050536 3834 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 14 FOLLOWER]: Snoozing failure detection for an additional: 0.549s I0529 14:48:57.050866 3834 raft_consensus.cc:273] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 14 FOLLOWER]: Starting election with quorum: opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } I0529 14:48:57.058217 3834 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 14 election: Requesting vote from peer aa0d62233d0a498886326226b4827f51 I0529 14:48:57.058993 3834 leader_election.cc:194] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 14 election: Requesting vote from peer 3b7563857bb84965989d41eebdd82d5f I0529 14:48:57.066854 3210 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 13 FOLLOWER]: Advancing to term 14 I0529 14:48:57.067667 2892 raft_consensus.cc:1246] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 LEADER]: Leader election vote request: Denying vote to candidate 1f8655f18f7e4c6691f4c12ebb22fd06 for term 14 because replica is either leader or believes a valid leader to be alive. I0529 14:48:57.069208 3210 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Snoozing failure detection for an additional: 0.903s I0529 14:48:57.069660 3499 leader_election.cc:327] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 14 election: Vote denied by peer 3b7563857bb84965989d41eebdd82d5f. Message: Invalid argument: T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 LEADER]: Leader election vote request: Denying vote to candidate 1f8655f18f7e4c6691f4c12ebb22fd06 for term 14 because replica is either leader or believes a valid leader to be alive. I0529 14:48:57.075757 3723 raft_consensus.cc:595] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 14 FOLLOWER]: Rejecting Update request from peer 3b7563857bb84965989d41eebdd82d5f for earlier term 13. Current term is 14. Ops: [] I0529 14:48:57.076205 3210 raft_consensus.cc:1261] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f8655f18f7e4c6691f4c12ebb22fd06 in term 14. I0529 14:48:57.078197 3505 leader_election.cc:313] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 14 election: Vote granted by peer aa0d62233d0a498886326226b4827f51 I0529 14:48:57.078213 3211 raft_consensus.cc:595] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Rejecting Update request from peer 3b7563857bb84965989d41eebdd82d5f for earlier term 13. Current term is 14. Ops: [] I0529 14:48:57.078472 3976 consensus_queue.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [LEADER]: Peer responded invalid term: Peer: 1f8655f18f7e4c6691f4c12ebb22fd06, Is new: false, Last received: 0.0, Next index: 26, Last known committed idx: 25, Last exchange result: ERROR I0529 14:48:57.078570 3505 leader_election.cc:214] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. I0529 14:48:57.079288 4004 raft_consensus.cc:1521] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 LEADER]: Stepping down as leader of term 13 I0529 14:48:57.079812 4004 raft_consensus.cc:360] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 3b7563857bb84965989d41eebdd82d5f, State: 1, Role: LEADER Watermarks: {Received: term: 13 index: 44 Committed: term: 2 index: 25} I0529 14:48:57.080387 3837 raft_consensus.cc:1478] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 14 FOLLOWER]: Snoozing failure detection for an additional: 0.875s I0529 14:48:57.080783 3837 raft_consensus.cc:1396] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 14 FOLLOWER]: Leader election won for term 14 I0529 14:48:57.080546 4004 consensus_queue.cc:117] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 2.25, Committed index: 2.25, Last appended: 13.44, Current term: 13, Majority size: 1, State: 1, Mode: NON_LEADER I0529 14:48:57.081254 3971 consensus_queue.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [NON_LEADER]: Peer responded invalid term: Peer: aa0d62233d0a498886326226b4827f51, Is new: false, Last received: 0.0, Next index: 26, Last known committed idx: 25, Last exchange result: ERROR I0529 14:48:57.082270 4004 consensus_peers.cc:239] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f: Closing peer: aa0d62233d0a498886326226b4827f51 I0529 14:48:57.083595 4004 consensus_peers.cc:239] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f: Closing peer: 1f8655f18f7e4c6691f4c12ebb22fd06 I0529 14:48:57.083557 3837 raft_consensus.cc:323] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 14 LEADER]: Becoming Leader. State: Replica: 1f8655f18f7e4c6691f4c12ebb22fd06, State: 1, Role: LEADER Watermarks: {Received: term: 9 index: 44 Committed: term: 2 index: 25} I0529 14:48:57.084724 4004 raft_consensus.cc:1528] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 13 FOLLOWER]: Advancing to term 14 I0529 14:48:57.084493 3837 consensus_queue.cc:109] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 2.25, Committed index: 2.25, Last appended: 9.44, Current term: 14, Majority size: 2, State: 1, Mode: LEADER I0529 14:48:57.098053 3690 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 6ms of latency in Log::Sync() I0529 14:48:57.102037 64262 catalog_manager.cc:1393] Tablet: 05b4fe7d1e414fb79249c14e6ae31236 reported consensus state change. New consensus state: current_term: 14 leader_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" quorum { opid_index: -1 local: false peers { permanent_uuid: "1f8655f18f7e4c6691f4c12ebb22fd06" member_type: VOTER last_known_addr { host: "127.245.143.2" port: 53754 } } peers { permanent_uuid: "aa0d62233d0a498886326226b4827f51" member_type: VOTER last_known_addr { host: "127.245.143.1" port: 43880 } } peers { permanent_uuid: "3b7563857bb84965989d41eebdd82d5f" member_type: VOTER last_known_addr { host: "127.245.143.0" port: 42480 } } } W0529 14:48:57.110383 4093 raft_consensus_state.cc:430] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 14 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 2 index: 25, New majority replicated is: term: 2 index: 25, Current term is: 14 INFO: Connected to the Master I0529 14:48:57.194566 3213 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Refusing update from remote peer 1f8655f18f7e4c6691f4c12ebb22fd06: Log matching property violated. Preceding OpId in replica: term: 5 index: 43. Preceding OpId from leader: term: 14 index: 45. (index mismatch) I0529 14:48:57.196678 2893 raft_consensus.cc:627] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 14 FOLLOWER]: Refusing update from remote peer 1f8655f18f7e4c6691f4c12ebb22fd06: Log matching property violated. Preceding OpId in replica: term: 13 index: 44. Preceding OpId from leader: term: 14 index: 45. (index mismatch) I0529 14:48:57.197785 4079 consensus_queue.cc:420] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: Connected to new peer: Peer: aa0d62233d0a498886326226b4827f51, Is new: false, Last received: 0.0, Next index: 26, Last known committed idx: 25, Last exchange result: ERROR I0529 14:48:57.198323 3837 consensus_queue.cc:420] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [LEADER]: Connected to new peer: Peer: 3b7563857bb84965989d41eebdd82d5f, Is new: false, Last received: 0.0, Next index: 26, Last known committed idx: 25, Last exchange result: ERROR I0529 14:48:57.202908 3837 log_cache.cc:278] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06: Successfully read 19 ops from disk. I0529 14:48:57.204335 4079 log_cache.cc:278] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06: Successfully read 19 ops from disk. I0529 14:48:57.210743 3215 raft_consensus.cc:573] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Deduplicated request from leader. Original: 2.25->[2.26-14.45] Dedup: 5.38->[6.39-14.45] I0529 14:48:57.210999 2894 raft_consensus.cc:573] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 14 FOLLOWER]: Deduplicated request from leader. Original: 2.25->[2.26-14.45] Dedup: 6.41->[9.42-14.45] I0529 14:48:57.211225 3215 raft_consensus_state.cc:340] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Aborting all transactions after (but not including): 38. Current State: Replica: aa0d62233d0a498886326226b4827f51, State: 1, Role: FOLLOWER Watermarks: {Received: term: 5 index: 43 Committed: term: 2 index: 25} I0529 14:48:57.211593 3215 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.39 I0529 14:48:57.211483 2894 raft_consensus_state.cc:340] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 14 FOLLOWER]: Aborting all transactions after (but not including): 41. Current State: Replica: 3b7563857bb84965989d41eebdd82d5f, State: 1, Role: FOLLOWER Watermarks: {Received: term: 13 index: 44 Committed: term: 2 index: 25} I0529 14:48:57.211894 2894 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 14 FOLLOWER]: Aborting uncommitted operation due to leader change: 6.42 I0529 14:48:57.212934 3215 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.40 I0529 14:48:57.213251 2894 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 14 FOLLOWER]: Aborting uncommitted operation due to leader change: 6.43 I0529 14:48:57.214095 2894 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f [term 14 FOLLOWER]: Aborting uncommitted operation due to leader change: 13.44 I0529 14:48:57.214102 3215 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.41 I0529 14:48:57.214300 2894 raft_consensus.cc:1423] T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f: NO_OP replication failed: Aborted: Transaction aborted by new leader I0529 14:48:57.215219 3215 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.42 I0529 14:48:57.216635 3215 raft_consensus_state.cc:366] T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 [term 14 FOLLOWER]: Aborting uncommitted operation due to leader change: 5.43 I0529 14:48:57.220878 2857 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 42ms of latency in Log::Sync() I0529 14:48:57.231158 3180 log.cc:547] T 05b4fe7d1e414fb79249c14e6ae31236: Injecting 25ms of latency in Log::Sync() INFO: Connected to all 3 Tablet Servers INFO: Table test-workload is HEALTHY INFO: The metadata for 1 tables is HEALTHY ----------------------- test-workload ----------------------- T 05b4fe7d1e414fb79249c14e6ae31236 P aa0d62233d0a498886326226b4827f51 (127.245.143.1:43880): Checksum: 53044265624 T 05b4fe7d1e414fb79249c14e6ae31236 P 3b7563857bb84965989d41eebdd82d5f (127.245.143.0:42480): Checksum: 53044265624 T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 (127.245.143.2:53754): Checksum: 53044265624 I0529 14:48:57.282768 4093 raft_consensus_state.cc:423] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06 [term 14 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 2 index: 25 New committed index is: term: 14 index: 45 src/kudu/integration-tests/cluster_verifier.cc:102: Failure Expected: (count) >= (expected_row_count), actual: 24 vs 25 src/kudu/integration-tests/raft_consensus-itest.cc:911: Failure Expected: v.CheckRowCount("test-workload", ClusterVerifier::AT_LEAST, workload.rows_inserted()) doesn't generate new fatal failures in the current thread. Actual: it does. I0529 14:48:57.356937 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-master with pid 64190 I0529 14:48:57.426506 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server with pid 2579 I0529 14:48:57.485401 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server with pid 2988 W0529 14:48:57.509809 3499 consensus_peers.cc:216] T 05b4fe7d1e414fb79249c14e6ae31236 P 1f8655f18f7e4c6691f4c12ebb22fd06: Couldn't send request to peer 3b7563857bb84965989d41eebdd82d5f for tablet 05b4fe7d1e414fb79249c14e6ae31236 Status: Network error: Client connection negotiation failed: client Connection to 127.245.143.0:42480: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0529 14:48:57.543021 3505 connection.cc:379] client Connection to 127.245.143.1:43880 recv error: Network error: recv error: Connection reset by peer (error 104) I0529 14:48:57.543218 62863 external_mini_cluster.cc:509] Killing /home/todd/run_tha_testHSjhNC/build/fastdebug/kudu-tablet_server with pid 3287 I0529 14:48:57.604737 62863 test_util.cc:56] ----------------------------------------------- I0529 14:48:57.604977 62863 test_util.cc:57] Had fatal failures, leaving test files at /tmp/kudutest-4099/raft_consensus-itest.RaftConsensusITest.InsertWithCrashyNodes.1432936117609050-62863 [ FAILED ] RaftConsensusITest.InsertWithCrashyNodes (14395 ms) [----------] 2 tests from RaftConsensusITest (19994 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (19995 ms total) [ PASSED ] 1 test. [ FAILED ] 1 test, listed below: [ FAILED ] RaftConsensusITest.InsertWithCrashyNodes 1 FAILED TEST