[==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from AdminCliTest [ RUN ] AdminCliTest.TestChangeConfig I0819 03:07:54.450461 24054 test_util.cc:118] Using random seed: 894153216 I0819 03:07:54.482003 24054 ts_itest-base.h:81] Starting cluster with: I0819 03:07:54.482048 24054 ts_itest-base.h:82] -------------- I0819 03:07:54.482059 24054 ts_itest-base.h:83] 3 tablet servers I0819 03:07:54.482077 24054 ts_itest-base.h:84] 2 replicas per TS I0819 03:07:54.482085 24054 ts_itest-base.h:85] -------------- I0819 03:07:54.483077 24054 external_mini_cluster.cc:521] Running /home/mpercy/src/kudu/build/fastdebug/kudu-master kudu-master --fs_wal_dir=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/master --fs_data_dirs=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/master --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/master --server_dump_info_path=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0819 03:07:54.613960 24319 mem_tracker.cc:98] MemTracker: hard memory limit is 12.356171 GB I0819 03:07:54.614310 24319 mem_tracker.cc:100] MemTracker: soft memory limit is 7.413702 GB I0819 03:07:54.620556 24319 master_main.cc:35] Initializing master server... I0819 03:07:54.623976 24319 server_base.cc:139] Could not load existing FS layout: Not found: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/master/instance: No such file or directory (error 2) I0819 03:07:54.624063 24319 server_base.cc:140] Creating new FS layout I0819 03:07:54.652765 24319 fs_manager.cc:315] Generated new instance metadata in path /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/master/instance: uuid: "f394f306d29643ceab45c8ab6419973d" format_stamp: "Formatted at 2015-08-19 10:07:54 on mpercy-ThinkPad-T530" I0819 03:07:54.740540 24319 fs_manager.cc:211] Opened local filesystem: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/master uuid: "f394f306d29643ceab45c8ab6419973d" format_stamp: "Formatted at 2015-08-19 10:07:54 on mpercy-ThinkPad-T530" I0819 03:07:54.759573 24319 hybrid_clock.cc:122] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 426835 I0819 03:07:54.760421 24319 master_main.cc:38] Starting Master server... I0819 03:07:54.925227 24319 rpc_server.cc:131] RPC server started. Bound to: 127.0.0.1:50464 I0819 03:07:54.925652 24319 webserver.cc:118] Starting webserver on localhost:0 I0819 03:07:54.925671 24319 webserver.cc:123] Document root: /home/mpercy/src/kudu/www I0819 03:07:54.926331 24319 webserver.cc:210] Webserver started. Bound to: http://127.0.0.1:45902/ I0819 03:07:54.926903 24319 server_base.cc:210] Dumped server information to /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/master/info.pb I0819 03:07:54.934278 24054 external_mini_cluster.cc:557] Started /home/mpercy/src/kudu/build/fastdebug/kudu-master as pid 24319 I0819 03:07:54.934710 24054 external_mini_cluster.cc:521] Running /home/mpercy/src/kudu/build/fastdebug/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-0 --fs_data_dirs=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-0 --rpc_bind_addresses=127.93.246.0:0 --local_ip_for_outbound_sockets=127.93.246.0 --webserver_interface=127.93.246.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:50464 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-0 --enable_leader_failure_detection=false --server_dump_info_path=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0819 03:07:54.960361 24640 tablet_bootstrap.cc:332] Tablet 00000000000000000000000000000000: Bootstrap starting. I0819 03:07:54.961664 24640 tablet_bootstrap.cc:496] Time spent opening tablet 00000000000000000000000000000000: real 0.000s user 0.000s sys 0.000s I0819 03:07:54.961866 24640 tablet_bootstrap.cc:431] No blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I0819 03:07:54.962692 24640 log.cc:288] Log is configured to *not* fsync() on all Append() calls I0819 03:07:54.973153 24640 tablet_bootstrap.cc:332] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I0819 03:07:54.974247 24640 local_consensus.cc:49] T 00000000000000000000000000000000 P f394f306d29643ceab45c8ab6419973d: Starting LocalConsensus... I0819 03:07:54.974427 24640 sys_catalog.cc:195] T 00000000000000000000000000000000 P f394f306d29643ceab45c8ab6419973d [sys.catalog]: SysCatalogTable state changed. New config config:opid_index: -1 local: true peers { permanent_uuid: "f394f306d29643ceab45c8ab6419973d" member_type: VOTER } I0819 03:07:54.974524 24640 sys_catalog.cc:198] T 00000000000000000000000000000000 P f394f306d29643ceab45c8ab6419973d [sys.catalog]: This master's current role is: LEADER, previous role was: FOLLOWER I0819 03:07:54.985077 24640 sys_catalog.cc:265] T 00000000000000000000000000000000 P f394f306d29643ceab45c8ab6419973d [sys.catalog]: configured and running, proceeding with master startup. I0819 03:07:54.993911 24319 master_main.cc:41] Master server successfully started. I0819 03:07:54.993968 24672 catalog_manager.cc:453] Loading table and tablet metadata into memory... I0819 03:07:55.120946 24642 mem_tracker.cc:98] MemTracker: hard memory limit is 12.356171 GB I0819 03:07:55.121279 24642 mem_tracker.cc:100] MemTracker: soft memory limit is 7.413702 GB I0819 03:07:55.159364 24642 tablet_server_main.cc:37] Initializing tablet server... I0819 03:07:55.169622 24642 server_base.cc:139] Could not load existing FS layout: Not found: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-0/instance: No such file or directory (error 2) I0819 03:07:55.169700 24642 server_base.cc:140] Creating new FS layout I0819 03:07:55.180953 24642 fs_manager.cc:315] Generated new instance metadata in path /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-0/instance: uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" format_stamp: "Formatted at 2015-08-19 10:07:55 on mpercy-ThinkPad-T530" I0819 03:07:55.210438 24642 fs_manager.cc:211] Opened local filesystem: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-0 uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" format_stamp: "Formatted at 2015-08-19 10:07:55 on mpercy-ThinkPad-T530" I0819 03:07:55.261771 24642 hybrid_clock.cc:122] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 427335 I0819 03:07:55.268604 24642 tablet_server_main.cc:40] Starting tablet server... I0819 03:07:55.709962 24642 rpc_server.cc:131] RPC server started. Bound to: 127.93.246.0:48612 I0819 03:07:55.710409 24642 webserver.cc:118] Starting webserver on 127.93.246.0:0 I0819 03:07:55.710510 24642 webserver.cc:123] Document root: /home/mpercy/src/kudu/www I0819 03:07:55.711076 24642 webserver.cc:210] Webserver started. Bound to: http://127.93.246.0:44555/ I0819 03:07:55.711607 24642 server_base.cc:210] Dumped server information to /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-0/info.pb I0819 03:07:55.716636 24642 tablet_server_main.cc:43] Tablet server successfully started. I0819 03:07:55.718181 24054 external_mini_cluster.cc:557] Started /home/mpercy/src/kudu/build/fastdebug/kudu-tserver as pid 24642 I0819 03:07:55.718471 24054 external_mini_cluster.cc:521] Running /home/mpercy/src/kudu/build/fastdebug/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1 --fs_data_dirs=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1 --rpc_bind_addresses=127.93.246.1:0 --local_ip_for_outbound_sockets=127.93.246.1 --webserver_interface=127.93.246.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:50464 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1 --enable_leader_failure_detection=false --server_dump_info_path=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0819 03:07:55.725878 25125 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:50464 I0819 03:07:55.726130 25125 heartbeater.cc:312] Registering TS with master... I0819 03:07:55.726349 25125 heartbeater.cc:318] Sending a full tablet report to master... I0819 03:07:55.727828 24537 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" instance_seqno: 1439978875637539 } with Master I0819 03:07:55.959272 25130 mem_tracker.cc:98] MemTracker: hard memory limit is 12.356171 GB I0819 03:07:55.959475 25130 mem_tracker.cc:100] MemTracker: soft memory limit is 7.413702 GB I0819 03:07:55.963551 25130 tablet_server_main.cc:37] Initializing tablet server... I0819 03:07:55.966853 25130 server_base.cc:139] Could not load existing FS layout: Not found: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/instance: No such file or directory (error 2) I0819 03:07:55.966989 25130 server_base.cc:140] Creating new FS layout I0819 03:07:55.980319 25130 fs_manager.cc:315] Generated new instance metadata in path /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/instance: uuid: "e93885ba47a54b9280807b4ea4803f83" format_stamp: "Formatted at 2015-08-19 10:07:55 on mpercy-ThinkPad-T530" I0819 03:07:56.019726 25130 fs_manager.cc:211] Opened local filesystem: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1 uuid: "e93885ba47a54b9280807b4ea4803f83" format_stamp: "Formatted at 2015-08-19 10:07:55 on mpercy-ThinkPad-T530" I0819 03:07:56.025063 25130 hybrid_clock.cc:122] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 427835 I0819 03:07:56.029659 25130 tablet_server_main.cc:40] Starting tablet server... I0819 03:07:56.208778 25130 rpc_server.cc:131] RPC server started. Bound to: 127.93.246.1:58332 I0819 03:07:56.209425 25130 webserver.cc:118] Starting webserver on 127.93.246.1:0 I0819 03:07:56.210294 25130 webserver.cc:123] Document root: /home/mpercy/src/kudu/www I0819 03:07:56.211163 25130 webserver.cc:210] Webserver started. Bound to: http://127.93.246.1:50535/ I0819 03:07:56.211844 25130 server_base.cc:210] Dumped server information to /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/info.pb I0819 03:07:56.213292 25130 tablet_server_main.cc:43] Tablet server successfully started. I0819 03:07:56.218480 24054 external_mini_cluster.cc:557] Started /home/mpercy/src/kudu/build/fastdebug/kudu-tserver as pid 25130 I0819 03:07:56.218745 24054 external_mini_cluster.cc:521] Running /home/mpercy/src/kudu/build/fastdebug/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-2 --fs_data_dirs=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-2 --rpc_bind_addresses=127.93.246.2:0 --local_ip_for_outbound_sockets=127.93.246.2 --webserver_interface=127.93.246.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:50464 --metrics_log_interval_ms=1000 --log_dir=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-2 --enable_leader_failure_detection=false --server_dump_info_path=/tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 I0819 03:07:56.219863 25541 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:50464 I0819 03:07:56.220053 25541 heartbeater.cc:312] Registering TS with master... I0819 03:07:56.220221 25541 heartbeater.cc:318] Sending a full tablet report to master... I0819 03:07:56.221103 24539 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698 } with Master I0819 03:07:56.422637 25558 mem_tracker.cc:98] MemTracker: hard memory limit is 12.356171 GB I0819 03:07:56.423863 25558 mem_tracker.cc:100] MemTracker: soft memory limit is 7.413702 GB I0819 03:07:56.455471 25558 tablet_server_main.cc:37] Initializing tablet server... I0819 03:07:56.465579 25558 server_base.cc:139] Could not load existing FS layout: Not found: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-2/instance: No such file or directory (error 2) I0819 03:07:56.466053 25558 server_base.cc:140] Creating new FS layout I0819 03:07:56.480170 25558 fs_manager.cc:315] Generated new instance metadata in path /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-2/instance: uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" format_stamp: "Formatted at 2015-08-19 10:07:56 on mpercy-ThinkPad-T530" I0819 03:07:56.518507 25558 fs_manager.cc:211] Opened local filesystem: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-2 uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" format_stamp: "Formatted at 2015-08-19 10:07:56 on mpercy-ThinkPad-T530" I0819 03:07:56.527169 25558 hybrid_clock.cc:122] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 427835 I0819 03:07:56.535321 25558 tablet_server_main.cc:40] Starting tablet server... I0819 03:07:56.691730 25558 rpc_server.cc:131] RPC server started. Bound to: 127.93.246.2:56584 I0819 03:07:56.692069 25558 webserver.cc:118] Starting webserver on 127.93.246.2:0 I0819 03:07:56.692085 25558 webserver.cc:123] Document root: /home/mpercy/src/kudu/www I0819 03:07:56.693027 25558 webserver.cc:210] Webserver started. Bound to: http://127.93.246.2:54522/ I0819 03:07:56.693691 25558 server_base.cc:210] Dumped server information to /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-2/info.pb I0819 03:07:56.700027 25810 heartbeater.cc:242] Connected to a leader master server at 127.0.0.1:50464 I0819 03:07:56.700181 25810 heartbeater.cc:312] Registering TS with master... I0819 03:07:56.700312 25810 heartbeater.cc:318] Sending a full tablet report to master... I0819 03:07:56.701093 24541 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" instance_seqno: 1439978876619398 } with Master I0819 03:07:56.703416 25558 tablet_server_main.cc:43] Tablet server successfully started. I0819 03:07:56.703927 24054 external_mini_cluster.cc:557] Started /home/mpercy/src/kudu/build/fastdebug/kudu-tserver as pid 25558 I0819 03:07:56.712230 24054 external_mini_cluster.cc:295] 3 TS(s) registered with Master W0819 03:07:56.713982 24318 reactor.cc:200] Client call kudu.master.MasterService.ListTabletServers has no timeout set for connection id: {remote=127.0.0.1:50464, user_credentials={real_user=mpercy, eff_user=}} I0819 03:07:56.725759 24549 catalog_manager.cc:660] CreateTable from {real_user=mpercy, eff_user=} at 127.0.0.1:41122: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 2 I0819 03:07:56.761302 25640 tablet_service.cc:450] Processing CreateTablet for tablet a688b3ea41fb40dc97dce27893bf018d (table=TestTable [id=6ed5a126dd3c4ad3ab2bdebb4552670b]), range=["", ""] I0819 03:07:56.762504 24835 tablet_service.cc:450] Processing CreateTablet for tablet a688b3ea41fb40dc97dce27893bf018d (table=TestTable [id=6ed5a126dd3c4ad3ab2bdebb4552670b]), range=["", ""] I0819 03:07:56.788589 25640 ts_tablet_manager.cc:656] Registered tablet a688b3ea41fb40dc97dce27893bf018d I0819 03:07:56.789101 25917 ts_tablet_manager.cc:530] Bootstrapping tablet: a688b3ea41fb40dc97dce27893bf018d I0819 03:07:56.790243 25917 tablet_bootstrap.cc:332] Tablet a688b3ea41fb40dc97dce27893bf018d: Bootstrap starting. I0819 03:07:56.791882 25917 tablet_bootstrap.cc:496] Time spent opening tablet a688b3ea41fb40dc97dce27893bf018d: real 0.001s user 0.000s sys 0.000s I0819 03:07:56.792208 25917 tablet_bootstrap.cc:431] No blocks or log segments found for tablet: a688b3ea41fb40dc97dce27893bf018d creating new one. I0819 03:07:56.792278 24835 ts_tablet_manager.cc:656] Registered tablet a688b3ea41fb40dc97dce27893bf018d I0819 03:07:56.793105 25917 log.cc:288] Log is configured to *not* fsync() on all Append() calls I0819 03:07:56.805819 25929 ts_tablet_manager.cc:530] Bootstrapping tablet: a688b3ea41fb40dc97dce27893bf018d I0819 03:07:56.807297 25929 tablet_bootstrap.cc:332] Tablet a688b3ea41fb40dc97dce27893bf018d: Bootstrap starting. I0819 03:07:56.808934 25929 tablet_bootstrap.cc:496] Time spent opening tablet a688b3ea41fb40dc97dce27893bf018d: real 0.001s user 0.000s sys 0.000s I0819 03:07:56.810585 25929 tablet_bootstrap.cc:431] No blocks or log segments found for tablet: a688b3ea41fb40dc97dce27893bf018d creating new one. I0819 03:07:56.813478 25929 log.cc:288] Log is configured to *not* fsync() on all Append() calls I0819 03:07:56.816848 25929 tablet_bootstrap.cc:332] Tablet a688b3ea41fb40dc97dce27893bf018d: No bootstrap required, opened a new log I0819 03:07:56.817163 25929 ts_tablet_manager.cc:535] Time spent bootstrapping tablet a688b3ea41fb40dc97dce27893bf018d: real 0.011s user 0.006s sys 0.000s I0819 03:07:56.819581 25917 tablet_bootstrap.cc:332] Tablet a688b3ea41fb40dc97dce27893bf018d: No bootstrap required, opened a new log I0819 03:07:56.819823 25917 ts_tablet_manager.cc:535] Time spent bootstrapping tablet a688b3ea41fb40dc97dce27893bf018d: real 0.031s user 0.011s sys 0.018s I0819 03:07:56.827217 25929 raft_consensus.cc:206] T a688b3ea41fb40dc97dce27893bf018d P 110c9f541b2b4d1fa1594ccbf83c1cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" member_type: VOTER last_known_addr { host: "127.93.246.2" port: 56584 } } peers { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" member_type: VOTER last_known_addr { host: "127.93.246.0" port: 48612 } } I0819 03:07:56.827746 25929 raft_consensus.cc:396] T a688b3ea41fb40dc97dce27893bf018d P 110c9f541b2b4d1fa1594ccbf83c1cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110c9f541b2b4d1fa1594ccbf83c1cd6, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0819 03:07:56.827628 25917 raft_consensus.cc:206] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 local: false peers { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" member_type: VOTER last_known_addr { host: "127.93.246.2" port: 56584 } } peers { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" member_type: VOTER last_known_addr { host: "127.93.246.0" port: 48612 } } I0819 03:07:56.827973 25917 raft_consensus.cc:396] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a722ac86bb42cc9a2142dd54e0dc53, State: 1, Role: FOLLOWER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0819 03:07:56.828027 25929 consensus_queue.cc:132] T a688b3ea41fb40dc97dce27893bf018d P 110c9f541b2b4d1fa1594ccbf83c1cd6 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0819 03:07:56.828234 25917 consensus_queue.cc:132] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [NON_LEADER]: queue going to NON_LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 0, Majority size: 1, State: 1, Mode: NON_LEADER I0819 03:07:56.829754 25929 ts_tablet_manager.cc:556] Time spent starting tablet a688b3ea41fb40dc97dce27893bf018d: real 0.012s user 0.004s sys 0.000s I0819 03:07:56.830097 25917 ts_tablet_manager.cc:556] Time spent starting tablet a688b3ea41fb40dc97dce27893bf018d: real 0.010s user 0.004s sys 0.000s I0819 03:07:56.839623 24054 external_mini_cluster.cc:295] 3 TS(s) registered with Master I0819 03:07:56.843775 25670 raft_consensus.cc:1699] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 0 FOLLOWER]: Advancing to term 1 I0819 03:07:56.844962 25670 raft_consensus.cc:299] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 1 FOLLOWER]: Starting election with config: opid_index: -1 local: false peers { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" member_type: VOTER last_known_addr { host: "127.93.246.2" port: 56584 } } peers { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" member_type: VOTER last_known_addr { host: "127.93.246.0" port: 48612 } } I0819 03:07:56.846220 25670 leader_election.cc:197] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [CANDIDATE]: Term 1 election: Requesting vote from peer 110c9f541b2b4d1fa1594ccbf83c1cd6 I0819 03:07:56.848763 24937 raft_consensus.cc:1699] T a688b3ea41fb40dc97dce27893bf018d P 110c9f541b2b4d1fa1594ccbf83c1cd6 [term 0 FOLLOWER]: Advancing to term 1 I0819 03:07:56.850641 24937 raft_consensus.cc:1386] T a688b3ea41fb40dc97dce27893bf018d P 110c9f541b2b4d1fa1594ccbf83c1cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a722ac86bb42cc9a2142dd54e0dc53 in term 1. I0819 03:07:56.851495 25613 leader_election.cc:317] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [CANDIDATE]: Term 1 election: Vote granted by peer 110c9f541b2b4d1fa1594ccbf83c1cd6 I0819 03:07:56.851594 25613 leader_election.cc:218] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0819 03:07:56.852016 25948 raft_consensus.cc:1561] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 1 FOLLOWER]: Leader election won for term 1 I0819 03:07:56.859777 25948 raft_consensus.cc:364] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 1 LEADER]: Becoming Leader. State: Replica: e0a722ac86bb42cc9a2142dd54e0dc53, State: 1, Role: LEADER Watermarks: {Received: term: 0 index: 0 Committed: term: 0 index: 0} I0819 03:07:56.859992 25948 consensus_queue.cc:124] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [LEADER]: queue going to LEADER mode. State: All replicated op: 0.0, Majority replicated op: 0.0, Committed index: 0.0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER W0819 03:07:56.871728 25971 raft_consensus_state.cc:487] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 1 LEADER]: Can't advance the committed index across term boundaries until operations from the current term are replicated. Last committed operation was: term: 0 index: 0, New majority replicated is: term: 0 index: 0, Current term is: 1 I0819 03:07:56.873330 24054 cluster_itest_util.cc:161] Not converged past 1 yet: 0.0 1.1 I0819 03:07:56.874197 24538 catalog_manager.cc:1395] Tablet: a688b3ea41fb40dc97dce27893bf018d reported consensus state change. New consensus state: current_term: 1 leader_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" config { opid_index: -1 local: false peers { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" member_type: VOTER last_known_addr { host: "127.93.246.2" port: 56584 } } peers { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" member_type: VOTER last_known_addr { host: "127.93.246.0" port: 48612 } } } I0819 03:07:56.975785 24054 cluster_itest_util.cc:161] Not converged past 1 yet: 0.0 1.1 I0819 03:07:57.177654 24054 cluster_itest_util.cc:161] Not converged past 1 yet: 0.0 1.1 I0819 03:07:57.364857 24941 raft_consensus.cc:672] T a688b3ea41fb40dc97dce27893bf018d P 110c9f541b2b4d1fa1594ccbf83c1cd6 [term 1 FOLLOWER]: Refusing update from remote peer e0a722ac86bb42cc9a2142dd54e0dc53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0819 03:07:57.366636 25948 consensus_queue.cc:493] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [LEADER]: Connected to new peer: Peer: 110c9f541b2b4d1fa1594ccbf83c1cd6, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: false I0819 03:07:57.375131 26271 raft_consensus_state.cc:480] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 1 LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I0819 03:07:57.485131 24054 test_workload.cc:146] TestWorkload: Skipping table creation because table TestTable already exists I0819 03:07:57.488116 24054 kudu-admin-test.cc:97] Adding tserver with uuid e93885ba47a54b9280807b4ea4803f83 as VOTER... I0819 03:07:57.488348 24054 kudu-admin-test.cc:104] Invoking command: /home/mpercy/src/kudu/build/fastdebug/kudu-admin -master_addresses 127.0.0.1:50464 change_config a688b3ea41fb40dc97dce27893bf018d ADD_SERVER e93885ba47a54b9280807b4ea4803f83 VOTER I0819 03:07:57.725045 25677 consensus_queue.cc:124] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [LEADER]: queue going to LEADER mode. State: All replicated op: 1.28, Majority replicated op: 1.28, Committed index: 1.28, Last appended: 1.28, Current term: 1, Majority size: 2, State: 1, Mode: LEADER I0819 03:07:57.725349 25677 consensus_peers.cc:272] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer 110c9f541b2b4d1fa1594ccbf83c1cd6 (127.93.246.0:48612): Closing peer: 110c9f541b2b4d1fa1594ccbf83c1cd6 I0819 03:07:57.728618 24952 raft_consensus.cc:672] T a688b3ea41fb40dc97dce27893bf018d P 110c9f541b2b4d1fa1594ccbf83c1cd6 [term 1 FOLLOWER]: Refusing update from remote peer e0a722ac86bb42cc9a2142dd54e0dc53: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 30. (index mismatch) I0819 03:07:57.729022 25948 consensus_queue.cc:493] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [LEADER]: Connected to new peer: Peer: 110c9f541b2b4d1fa1594ccbf83c1cd6, Is new: false, Last received: 1.28, Next index: 29, Last known committed idx: 27, Last exchange result: ERROR, Needs remote bootstrap: false I0819 03:07:57.731220 26540 consensus_queue.cc:424] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [LEADER]: Marked peer as needing remote bootstrap: Peer: e93885ba47a54b9280807b4ea4803f83, Is new: true, Last received: 0.0, Next index: 29, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: true I0819 03:07:57.731475 26271 raft_consensus_state.cc:547] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [term 1 LEADER]: Committing config change with OpId 1.29. Old config: { opid_index: -1 local: false peers { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" member_type: VOTER last_known_addr { host: "127.93.246.2" port: 56584 } } peers { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" member_type: VOTER last_known_addr { host: "127.93.246.0" port: 48612 } } }. New config: {opid_index: 29 local: false peers { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" member_type: VOTER last_known_addr { host: "127.93.246.2" port: 56584 } } peers { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" member_type: VOTER last_known_addr { host: "127.93.246.0" port: 48612 } } peers { permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" member_type: VOTER last_known_addr { host: "127.93.246.1" port: 58332 } }} I0819 03:07:57.732144 26540 consensus_queue.cc:254] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 [LEADER]: Peer needs remote bootstrap: Peer: e93885ba47a54b9280807b4ea4803f83, Is new: true, Last received: 0.0, Next index: 29, Last known committed idx: 0, Last exchange result: ERROR, Needs remote bootstrap: true I0819 03:07:57.733525 26540 consensus_peers.cc:224] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Sending request to remotely bootstrap I0819 03:07:57.734287 25393 ts_tablet_manager.cc:382] T a688b3ea41fb40dc97dce27893bf018d P e93885ba47a54b9280807b4ea4803f83: Initiating remote bootstrap from Peer e0a722ac86bb42cc9a2142dd54e0dc53 (127.93.246.2:56584) I0819 03:07:57.734385 25393 remote_bootstrap_client.cc:129] T a688b3ea41fb40dc97dce27893bf018d P e93885ba47a54b9280807b4ea4803f83: Remote bootstrap client: Beginning remote bootstrap session from remote peer at address 127.93.246.2:56584 I0819 03:07:57.735860 25698 remote_bootstrap_service.cc:110] Beginning new remote bootstrap session on tablet a688b3ea41fb40dc97dce27893bf018d from peer e93885ba47a54b9280807b4ea4803f83 at {real_user=mpercy, eff_user=} at 127.93.246.1:39031: session id = e93885ba47a54b9280807b4ea4803f83-a688b3ea41fb40dc97dce27893bf018d I0819 03:07:57.737412 24954 raft_consensus_state.cc:547] T a688b3ea41fb40dc97dce27893bf018d P 110c9f541b2b4d1fa1594ccbf83c1cd6 [term 1 FOLLOWER]: Committing config change with OpId 1.29. Old config: { opid_index: -1 local: false peers { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" member_type: VOTER last_known_addr { host: "127.93.246.2" port: 56584 } } peers { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" member_type: VOTER last_known_addr { host: "127.93.246.0" port: 48612 } } }. New config: {opid_index: 29 local: false peers { permanent_uuid: "e0a722ac86bb42cc9a2142dd54e0dc53" member_type: VOTER last_known_addr { host: "127.93.246.2" port: 56584 } } peers { permanent_uuid: "110c9f541b2b4d1fa1594ccbf83c1cd6" member_type: VOTER last_known_addr { host: "127.93.246.0" port: 48612 } } peers { permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" member_type: VOTER last_known_addr { host: "127.93.246.1" port: 58332 } }} I0819 03:07:57.750630 25393 ts_tablet_manager.cc:656] Registered tablet a688b3ea41fb40dc97dce27893bf018d I0819 03:07:57.750828 25393 remote_bootstrap_client.cc:339] T a688b3ea41fb40dc97dce27893bf018d P e93885ba47a54b9280807b4ea4803f83: Remote bootstrap client: Starting download of 0 data blocks... I0819 03:07:57.755372 25393 remote_bootstrap_client.cc:304] T a688b3ea41fb40dc97dce27893bf018d P e93885ba47a54b9280807b4ea4803f83: Remote bootstrap client: Starting download of 1 WAL segments... I0819 03:07:57.755643 25393 tablet_bootstrap.cc:332] Tablet a688b3ea41fb40dc97dce27893bf018d: RemoteBootstrap: Downloading WAL segment with seq. number 1 (1/1) I0819 03:07:58.211305 25393 remote_bootstrap_client.cc:235] T a688b3ea41fb40dc97dce27893bf018d P e93885ba47a54b9280807b4ea4803f83: Remote bootstrap client: Remote bootstrap complete. Replacing tablet superblock. I0819 03:07:58.211335 25393 tablet_bootstrap.cc:332] Tablet a688b3ea41fb40dc97dce27893bf018d: RemoteBootstrap: Replacing tablet superblock I0819 03:07:58.232641 25711 remote_bootstrap_service.cc:231] Request end of remote bootstrap session e93885ba47a54b9280807b4ea4803f83-a688b3ea41fb40dc97dce27893bf018d received from {real_user=mpercy, eff_user=} at 127.93.246.1:39031 I0819 03:07:58.232748 25711 remote_bootstrap_service.cc:312] Ending remote bootstrap session e93885ba47a54b9280807b4ea4803f83-a688b3ea41fb40dc97dce27893bf018d on tablet a688b3ea41fb40dc97dce27893bf018d with peer e93885ba47a54b9280807b4ea4803f83 I0819 03:07:58.234093 26826 ts_tablet_manager.cc:530] Bootstrapping tablet: a688b3ea41fb40dc97dce27893bf018d I0819 03:07:58.237385 26826 tablet_bootstrap.cc:332] Tablet a688b3ea41fb40dc97dce27893bf018d: Bootstrap starting. I0819 03:07:58.239100 26826 tablet_bootstrap.cc:496] Time spent opening tablet a688b3ea41fb40dc97dce27893bf018d: real 0.001s user 0.000s sys 0.000s I0819 03:07:58.239428 26826 tablet_bootstrap.cc:573] Will attempt to recover log segment: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d/wal-000000001 to: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d.recovery/wal-000000001 I0819 03:07:58.239498 26826 tablet_bootstrap.cc:583] Moved log directory: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d to recovery directory: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d.recovery W0819 03:07:58.240139 26826 log_util.cc:149] Could not read footer for segment: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d.recovery/wal-000000001. Status: Not found: Footer not found. Footer magic doesn't match W0819 03:07:58.240238 26826 log_reader.cc:138] Segment: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d.recovery/wal-000000001 was likely left in-progress after a previous crash. Will try to rebuild footer by scanning data I0819 03:07:58.241868 26826 log_util.cc:498] Scanning /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d.recovery/wal-000000001 for valid entry headers following offset 7261... I0819 03:07:58.251670 24054 kudu-admin-test.cc:119] Waiting for replicas to agree... I0819 03:07:58.255537 24054 cluster_itest_util.cc:161] Not converged past 33 yet: 1.33 1.33 I0819 03:07:58.266154 26826 log_util.cc:535] Found no log entry headers I0819 03:07:58.266242 26826 log_util.cc:465] Ignoring corruption Corruption: CRC mismatch in log header: Log file corrupted. Failed trying to read batch #66 at offset 7261 for segment at /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d.recovery/wal-000000001. Prior batch offsets: 6863 7045 7079 7261: there were no further log entries following this one, so likely the log was truncated during writing I0819 03:07:58.266366 26826 log_util.cc:203] Successfully rebuilt footer for segment: /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054/raft_consensus-itest-cluster/ts-1/wals/a688b3ea41fb40dc97dce27893bf018d.recovery/wal-000000001 (valid entries through byte offset 7261) I0819 03:07:58.267228 26826 tablet.cc:732] Rewinding schema during bootstrap to Schema [ 10:key[int32 NOT NULL], 11:int_val[int32 NOT NULL], 12:string_val[string NULLABLE] ] I0819 03:07:58.268695 26826 log.cc:288] Log is configured to *not* fsync() on all Append() calls F0819 03:07:58.292922 26826 mvcc.cc:512] Check failed: _s.ok() Bad status: Illegal state: Timestamp: 5898153483160805376 is already committed. Current Snapshot: MvccSnapshot[committed={T|T < 5898153483161358336}] *** Check failure stack trace: *** @ 0x7f0aec92649d google::LogMessage::Fail() at ??:0 @ 0x7f0aec928449 google::LogMessage::SendToLog() at ??:0 @ 0x7f0aec926062 google::LogMessage::Flush() at ??:0 @ 0x7f0aec928ddf google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f0af2852eb0 kudu::tablet::ScopedTransaction::ScopedTransaction() at ??:0 @ 0x7f0af26c4b7f kudu::tablet::Tablet::StartTransaction() at ??:0 @ 0x7f0af2738041 kudu::tablet::TabletBootstrap::PlayWriteRequest() at ??:0 @ 0x7f0af2737316 kudu::tablet::TabletBootstrap::HandleEntryPair() at ??:0 @ 0x7f0af2735535 kudu::tablet::TabletBootstrap::ApplyCommitMessage() at ??:0 @ 0x7f0af27342a9 kudu::tablet::TabletBootstrap::HandleCommitMessage() at ??:0 @ 0x7f0af2733214 kudu::tablet::TabletBootstrap::HandleEntry() at ??:0 @ 0x7f0af272f97e kudu::tablet::TabletBootstrap::PlaySegments() at ??:0 @ 0x7f0af272c95a kudu::tablet::TabletBootstrap::Bootstrap() at ??:0 @ 0x7f0af272bc65 kudu::tablet::BootstrapTablet() at ??:0 @ 0x7f0af42ef855 kudu::tserver::TSTabletManager::OpenTablet() at ??:0 @ 0x7f0af43034be boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f0af0f0917f boost::function0<>::operator()() at ??:0 @ 0x7f0aed7d2470 kudu::ThreadPool::DispatchThread() at ??:0 @ 0x7f0aed7d888e boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f0af0f0917f boost::function0<>::operator()() at ??:0 @ 0x7f0aed7bd782 kudu::Thread::SuperviseThread() at ??:0 @ 0x7f0aeeed4182 start_thread at ??:0 @ 0x7f0aeb9d847d (unknown) at ??:0 @ (nil) (unknown) W0819 03:07:58.543999 24315 connection.cc:379] client Connection to 127.93.246.1:58332 recv error: Network error: recv error: Connection reset by peer (error 104) W0819 03:07:58.549046 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: recv error: Connection reset by peer (error 104) I0819 03:07:58.549096 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:07:58.727402 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0819 03:07:58.749938 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:07:58.749969 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:07:59.050815 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:07:59.050863 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:07:59.227176 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0819 03:07:59.451576 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:07:59.451620 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:07:59.727574 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0819 03:07:59.952536 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:07:59.952579 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:00.227661 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0819 03:08:00.553414 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:00.553462 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:00.728018 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0819 03:08:01.228060 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0819 03:08:01.254426 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:01.254472 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:01.728071 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0819 03:08:02.059439 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:02.059492 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:02.229425 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0819 03:08:02.736099 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0819 03:08:02.972743 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:02.972786 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:03.232096 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0819 03:08:03.737607 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. W0819 03:08:03.973717 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:03.973764 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:04.235288 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. W0819 03:08:04.735265 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 13 times. W0819 03:08:05.000751 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:05.000803 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:05.244588 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 14 times. W0819 03:08:05.738636 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 15 times. W0819 03:08:06.018628 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:06.018676 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:06.238903 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 16 times. W0819 03:08:06.771471 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 17 times. W0819 03:08:07.022390 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:07.022438 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:07.244848 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 18 times. W0819 03:08:07.742296 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 19 times. W0819 03:08:08.025435 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:08.025485 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:08.253597 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 20 times. W0819 03:08:08.742794 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 21 times. W0819 03:08:09.027099 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:09.027146 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:09.243811 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 22 times. W0819 03:08:09.743077 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 23 times. W0819 03:08:10.028039 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:10.028085 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:10.258977 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 24 times. W0819 03:08:10.743381 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 25 times. W0819 03:08:11.035325 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:11.035373 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:11.303916 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 26 times. W0819 03:08:11.746297 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 27 times. W0819 03:08:12.036411 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:12.036458 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:12.274410 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 28 times. W0819 03:08:12.746479 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 29 times. W0819 03:08:13.037274 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:13.037314 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:13.303735 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 30 times. W0819 03:08:13.748661 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 31 times. W0819 03:08:14.042948 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:14.043012 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:14.248031 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 32 times. W0819 03:08:14.757055 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 33 times. W0819 03:08:15.076467 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:15.076527 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:15.250030 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 34 times. W0819 03:08:15.750365 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 35 times. W0819 03:08:16.077450 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:16.077505 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:16.254901 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 36 times. W0819 03:08:16.748810 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 37 times. W0819 03:08:17.078311 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:17.078349 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:17.248988 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 38 times. W0819 03:08:17.749161 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 39 times. W0819 03:08:18.087589 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:18.087630 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:18.251441 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 40 times. W0819 03:08:18.749703 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 41 times. W0819 03:08:19.088608 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:19.088649 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:19.249657 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 42 times. W0819 03:08:19.750097 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 43 times. W0819 03:08:20.089499 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:20.089537 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:20.250012 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 44 times. W0819 03:08:20.750455 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 45 times. W0819 03:08:21.090329 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:21.090379 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:21.250568 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 46 times. W0819 03:08:21.750721 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 47 times. W0819 03:08:22.091419 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:22.091476 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:22.251168 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 48 times. W0819 03:08:22.751245 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 49 times. W0819 03:08:23.092349 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:23.092398 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:23.251231 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 50 times. W0819 03:08:23.751384 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 51 times. W0819 03:08:24.093803 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:24.093852 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:24.254760 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 52 times. W0819 03:08:24.751899 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 53 times. W0819 03:08:25.095113 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:25.095191 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:25.252341 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 54 times. W0819 03:08:25.758126 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 55 times. W0819 03:08:26.111613 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:26.111654 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:26.257452 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 56 times. W0819 03:08:26.760175 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 57 times. W0819 03:08:27.113051 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:27.113097 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:27.257724 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 58 times. W0819 03:08:27.757889 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 59 times. W0819 03:08:28.117898 24054 cluster_itest_util.cc:158] Got error getting last opid for each replica: Network error: Failed to fetch last op id from permanent_uuid: "e93885ba47a54b9280807b4ea4803f83" instance_seqno: 1439978876166698: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111) I0819 03:08:28.117938 24054 cluster_itest_util.cc:161] Not converged past 33 yet: W0819 03:08:28.258574 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 60 times. W0819 03:08:28.758316 25615 consensus_peers.cc:247] T a688b3ea41fb40dc97dce27893bf018d P e0a722ac86bb42cc9a2142dd54e0dc53 -> Peer e93885ba47a54b9280807b4ea4803f83 (127.93.246.1:58332): Couldn't send request to peer e93885ba47a54b9280807b4ea4803f83 for tablet a688b3ea41fb40dc97dce27893bf018d Status: Network error: Client connection negotiation failed: client Connection to 127.93.246.1:58332: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 61 times. src/kudu/tools/kudu-admin-test.cc:127: Failure Failed Bad status: Timed out: Index 33 not available on all replicas after 30.000s. I0819 03:08:29.118755 24054 external_mini_cluster.cc:610] Killing /home/mpercy/src/kudu/build/fastdebug/kudu-master with pid 24319 I0819 03:08:29.136703 24054 external_mini_cluster.cc:610] Killing /home/mpercy/src/kudu/build/fastdebug/kudu-tserver with pid 24642 I0819 03:08:29.154407 24054 external_mini_cluster.cc:610] Killing /home/mpercy/src/kudu/build/fastdebug/kudu-tserver with pid 25558 I0819 03:08:29.191514 24054 test_util.cc:56] ----------------------------------------------- I0819 03:08:29.191659 24054 test_util.cc:57] Had fatal failures, leaving test files at /tmp/kudutest-1000/kudu-admin-test.AdminCliTest.TestChangeConfig.1439978874446891-24054 [ FAILED ] AdminCliTest.TestChangeConfig (34744 ms) [----------] 1 test from AdminCliTest (34744 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (34744 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] AdminCliTest.TestChangeConfig 1 FAILED TEST