[==========] Running 8 tests from 1 test case. [----------] Global test environment set-up. [----------] 8 tests from DistConsensusTest [ RUN ] DistConsensusTest.TestGetPermanentUuid WARNING: Logging before InitGoogleLogging() is written to STDERR I1103 05:09:45.633924 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master kudu-master --master_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --master_web_port=0 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:45.654680 31089 master_main.cc:27] Initializing master server... I1103 05:09:46.346648 31089 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/master/instance: uuid: "91232a4c9a7f47ca9e168979f7167d11" format_stamp: "Formatted at 2014-11-03 13:09:45 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:46.346771 31089 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/master uuid: "91232a4c9a7f47ca9e168979f7167d11" format_stamp: "Formatted at 2014-11-03 13:09:45 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:46.350653 31089 master_main.cc:30] Starting master server... I1103 05:09:46.356389 31089 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:46.356426 31089 webserver.cc:117] Document root disabled I1103 05:09:46.356952 31089 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:33765 I1103 05:09:46.357185 31089 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb I1103 05:09:46.364998 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:50769 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:46.381525 31624 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:46.436276 31624 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/instance: uuid: "2d95119901b144a8946d907e7d644703" format_stamp: "Formatted at 2014-11-03 13:09:46 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:46.438968 31624 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 uuid: "2d95119901b144a8946d907e7d644703" format_stamp: "Formatted at 2014-11-03 13:09:46 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:46.442733 31624 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:46.445981 31624 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:46.446012 31624 webserver.cc:117] Document root disabled I1103 05:09:46.446449 31624 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:35373 I1103 05:09:46.449389 31624 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb I1103 05:09:46.449920 31624 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:46.450044 31624 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:46.452210 31779 heartbeater.cc:227] Registering TS with master... I1103 05:09:46.452246 31779 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:46.457548 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:50769 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:46.475337 31821 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:46.540338 31610 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: Bootstrap starting. I1103 05:09:46.540808 31610 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one. I1103 05:09:46.541107 31610 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:09:46.541332 31880 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:09:46.541627 31610 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: No bootstrap required, opened a new log I1103 05:09:46.542765 31886 tablet_peer.cc:187] Configuration changed for tablet: 11111111111111111111111111111111 in peer: 91232a4c9a7f47ca9e168979f7167d11 Changed from: seqno: 0 local: true peers { permanent_uuid: "91232a4c9a7f47ca9e168979f7167d11" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "91232a4c9a7f47ca9e168979f7167d11" role: LEADER } I1103 05:09:46.542803 31886 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "91232a4c9a7f47ca9e168979f7167d11" role: LEADER } I1103 05:09:46.571766 31821 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/instance: uuid: "8e4eb198ffc048b4b735f1e6cccd2a28" format_stamp: "Formatted at 2014-11-03 13:09:46 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:46.571905 31821 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 uuid: "8e4eb198ffc048b4b735f1e6cccd2a28" format_stamp: "Formatted at 2014-11-03 13:09:46 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:46.575747 31821 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:46.578708 31821 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:46.578740 31821 webserver.cc:117] Document root disabled I1103 05:09:46.579285 31821 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:37412 I1103 05:09:46.579725 31821 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb I1103 05:09:46.580505 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:50769 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:46.581719 31821 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:46.581990 31821 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:46.583293 31957 heartbeater.cc:227] Registering TS with master... I1103 05:09:46.583343 31957 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:46.601673 31958 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:46.645572 31958 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/instance: uuid: "1385df2a9a594252a864775ae89ce418" format_stamp: "Formatted at 2014-11-03 13:09:46 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:46.645689 31958 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 uuid: "1385df2a9a594252a864775ae89ce418" format_stamp: "Formatted at 2014-11-03 13:09:46 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:46.648715 31958 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:46.652753 31958 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:46.652767 31958 webserver.cc:117] Document root disabled I1103 05:09:46.653249 31958 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:45789 I1103 05:09:46.653425 31958 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestGetPermanentUuid.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb I1103 05:09:46.653632 31958 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:46.653745 31958 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:46.655591 32104 heartbeater.cc:227] Registering TS with master... I1103 05:09:46.655628 32104 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:46.659216 31610 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: Bootstrap starting. I1103 05:09:46.659690 31610 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I1103 05:09:46.660439 31610 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I1103 05:09:46.660902 31610 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. W1103 05:09:46.661155 31539 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:09:46.661144 31513 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. I1103 05:09:46.661211 32121 tablet_peer.cc:187] Configuration changed for tablet: 00000000000000000000000000000000 in peer: 91232a4c9a7f47ca9e168979f7167d11 Changed from: seqno: 0 local: true peers { permanent_uuid: "91232a4c9a7f47ca9e168979f7167d11" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "91232a4c9a7f47ca9e168979f7167d11" role: LEADER } I1103 05:09:46.661242 32121 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "91232a4c9a7f47ca9e168979f7167d11" role: LEADER } I1103 05:09:46.661252 31610 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:09:46.661427 31089 master_main.cc:33] Master server successfully started. I1103 05:09:46.661823 31537 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "8e4eb198ffc048b4b735f1e6cccd2a28" instance_seqno: 1415020186578544 } with Master I1103 05:09:47.661711 31537 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "1385df2a9a594252a864775ae89ce418" instance_seqno: 1415020186652620 } as {real_user=jenkins, eff_user=} at 127.0.0.1:44681; Asking this server to re-register. I1103 05:09:47.661711 31513 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "2d95119901b144a8946d907e7d644703" instance_seqno: 1415020186445839 } as {real_user=jenkins, eff_user=} at 127.0.0.1:44664; Asking this server to re-register. I1103 05:09:47.662039 31779 heartbeater.cc:227] Registering TS with master... I1103 05:09:47.662045 32104 heartbeater.cc:227] Registering TS with master... I1103 05:09:47.662070 31779 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:47.662070 32104 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:47.662220 31555 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "2d95119901b144a8946d907e7d644703" instance_seqno: 1415020186445839 } with Master I1103 05:09:47.662263 31547 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "1385df2a9a594252a864775ae89ce418" instance_seqno: 1415020186652620 } with Master I1103 05:09:47.662829 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:09:47.663120 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:09:47.666578 31538 catalog_manager.cc:437] CreateTable from {real_user=jenkins, eff_user=} at 127.0.0.1:44723: name: "TestTable" schema { columns { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I1103 05:09:47.668987 31757 tablet_service.cc:264] Processing CreateTablet for tablet c2927cf900a0421394b3235d03330379 (table=TestTable [id=0986ead88c934e7cb67d53fe0005fd3b]), range=["", ""] I1103 05:09:47.669116 31934 tablet_service.cc:264] Processing CreateTablet for tablet c2927cf900a0421394b3235d03330379 (table=TestTable [id=0986ead88c934e7cb67d53fe0005fd3b]), range=["", ""] I1103 05:09:47.669518 32077 tablet_service.cc:264] Processing CreateTablet for tablet c2927cf900a0421394b3235d03330379 (table=TestTable [id=0986ead88c934e7cb67d53fe0005fd3b]), range=["", ""] I1103 05:09:47.842272 31934 ts_tablet_manager.cc:464] Registered tablet c2927cf900a0421394b3235d03330379 I1103 05:09:47.842308 32077 ts_tablet_manager.cc:464] Registered tablet c2927cf900a0421394b3235d03330379 I1103 05:09:47.842430 1202 ts_tablet_manager.cc:325] Bootstrapping tablet: c2927cf900a0421394b3235d03330379 I1103 05:09:47.842552 1201 ts_tablet_manager.cc:325] Bootstrapping tablet: c2927cf900a0421394b3235d03330379 I1103 05:09:47.842700 1202 tablet_bootstrap.cc:234] Tablet c2927cf900a0421394b3235d03330379: Bootstrap starting. I1103 05:09:47.842715 1201 tablet_bootstrap.cc:234] Tablet c2927cf900a0421394b3235d03330379: Bootstrap starting. I1103 05:09:47.843257 1202 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: c2927cf900a0421394b3235d03330379 creating new one. I1103 05:09:47.843273 1201 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: c2927cf900a0421394b3235d03330379 creating new one. I1103 05:09:47.843526 1202 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:09:47.843690 1201 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:09:47.843906 1211 env_posix.cc:511] The filesystem does not support fallocate(). W1103 05:09:47.844218 1212 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:09:47.844321 1202 tablet_bootstrap.cc:234] Tablet c2927cf900a0421394b3235d03330379: No bootstrap required, opened a new log I1103 05:09:47.844354 1202 ts_tablet_manager.cc:330] Time spent tablet c2927cf900a0421394b3235d03330379 bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:09:47.844707 1202 raft_consensus.cc:102] T c2927cf900a0421394b3235d03330379 P 1385df2a9a594252a864775ae89ce418 [CANDIDATE]: Replica starting. Triggering 0 pending transactions. I1103 05:09:47.844739 1201 tablet_bootstrap.cc:234] Tablet c2927cf900a0421394b3235d03330379: No bootstrap required, opened a new log I1103 05:09:47.844770 1201 ts_tablet_manager.cc:330] Time spent tablet c2927cf900a0421394b3235d03330379 bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:09:47.844977 1202 raft_consensus.cc:235] T c2927cf900a0421394b3235d03330379 P 1385df2a9a594252a864775ae89ce418 [LEADER]: Becoming Leader I1103 05:09:47.845077 1201 raft_consensus.cc:102] T c2927cf900a0421394b3235d03330379 P 8e4eb198ffc048b4b735f1e6cccd2a28 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:09:47.845237 1201 raft_consensus.cc:302] T c2927cf900a0421394b3235d03330379 P 8e4eb198ffc048b4b735f1e6cccd2a28 [FOLLOWER]: Becoming Follower/Learner I1103 05:09:47.845299 1201 ts_tablet_manager.cc:350] Time spent starting tablet c2927cf900a0421394b3235d03330379: real 0.001s user 0.000s sys 0.000s I1103 05:09:47.846472 1202 ts_tablet_manager.cc:350] Time spent starting tablet c2927cf900a0421394b3235d03330379: real 0.002s user 0.002s sys 0.000s I1103 05:09:47.903024 31757 ts_tablet_manager.cc:464] Registered tablet c2927cf900a0421394b3235d03330379 I1103 05:09:47.903594 1284 ts_tablet_manager.cc:325] Bootstrapping tablet: c2927cf900a0421394b3235d03330379 I1103 05:09:47.903719 1284 tablet_bootstrap.cc:234] Tablet c2927cf900a0421394b3235d03330379: Bootstrap starting. I1103 05:09:47.904124 1284 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: c2927cf900a0421394b3235d03330379 creating new one. I1103 05:09:47.904477 1284 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:09:47.904886 1289 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:09:47.905277 1284 tablet_bootstrap.cc:234] Tablet c2927cf900a0421394b3235d03330379: No bootstrap required, opened a new log I1103 05:09:47.905324 1284 ts_tablet_manager.cc:330] Time spent tablet c2927cf900a0421394b3235d03330379 bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:09:47.905720 1284 raft_consensus.cc:102] T c2927cf900a0421394b3235d03330379 P 2d95119901b144a8946d907e7d644703 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:09:47.905836 1284 raft_consensus.cc:302] T c2927cf900a0421394b3235d03330379 P 2d95119901b144a8946d907e7d644703 [FOLLOWER]: Becoming Follower/Learner I1103 05:09:47.905900 1284 ts_tablet_manager.cc:350] Time spent starting tablet c2927cf900a0421394b3235d03330379: real 0.001s user 0.000s sys 0.000s W1103 05:09:48.345495 1217 raft_consensus_state.cc:463] T c2927cf900a0421394b3235d03330379 P 1385df2a9a594252a864775ae89ce418 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 W1103 05:09:48.345775 1213 raft_consensus_state.cc:463] T c2927cf900a0421394b3235d03330379 P 1385df2a9a594252a864775ae89ce418 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:09:48.347136 31767 raft_consensus.cc:608] T c2927cf900a0421394b3235d03330379 P 2d95119901b144a8946d907e7d644703 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:09:48.347272 32044 consensus_queue.cc:451] Connected to peer: 2d95119901b144a8946d907e7d644703 whose last received watermark is: term: 0 index: 0 I1103 05:09:48.347589 31944 raft_consensus.cc:608] T c2927cf900a0421394b3235d03330379 P 8e4eb198ffc048b4b735f1e6cccd2a28 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:09:48.347719 32046 consensus_queue.cc:451] Connected to peer: 8e4eb198ffc048b4b735f1e6cccd2a28 whose last received watermark is: term: 0 index: 0 I1103 05:09:48.348193 32044 raft_consensus_state.cc:456] T c2927cf900a0421394b3235d03330379 P 1385df2a9a594252a864775ae89ce418 [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I1103 05:09:48.348284 1213 tablet_peer.cc:187] Configuration changed for tablet: c2927cf900a0421394b3235d03330379 in peer: 1385df2a9a594252a864775ae89ce418 Changed from: seqno: -1 local: false peers { permanent_uuid: "2d95119901b144a8946d907e7d644703" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 52124 } } peers { permanent_uuid: "8e4eb198ffc048b4b735f1e6cccd2a28" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55511 } } peers { permanent_uuid: "1385df2a9a594252a864775ae89ce418" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 37579 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "2d95119901b144a8946d907e7d644703" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 52124 } } peers { permanent_uuid: "8e4eb198ffc048b4b735f1e6cccd2a28" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55511 } } peers { permanent_uuid: "1385df2a9a594252a864775ae89ce418" role: LEADER last_known_addr { host: "127.0.0.1" port: 37579 } } I1103 05:09:48.662729 31556 catalog_manager.cc:1127] Tablet: c2927cf900a0421394b3235d03330379 reported quorum change. New quorum: seqno: 0 local: false peers { permanent_uuid: "2d95119901b144a8946d907e7d644703" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 52124 } } peers { permanent_uuid: "8e4eb198ffc048b4b735f1e6cccd2a28" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55511 } } peers { permanent_uuid: "1385df2a9a594252a864775ae89ce418" role: LEADER last_known_addr { host: "127.0.0.1" port: 37579 } } I1103 05:09:48.729127 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master W1103 05:09:48.729228 31086 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:50769, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:48.730123 2170 reactor.cc:199] Client call GetNodeInstance has no timeout set for connection id: {remote=127.0.0.1:37579, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:09:48.731086 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master with pid 31089 I1103 05:09:48.733383 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 31624 I1103 05:09:48.735265 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 31821 I1103 05:09:48.737094 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 31958 [ OK ] DistConsensusTest.TestGetPermanentUuid (3113 ms) [ RUN ] DistConsensusTest.TestInsertAndMutateThroughConsensus I1103 05:09:48.743198 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master kudu-master --master_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --master_web_port=0 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:48.759593 2173 master_main.cc:27] Initializing master server... I1103 05:09:48.811203 2173 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master/instance: uuid: "2facf84ee8b640bb8e539aa686f8039c" format_stamp: "Formatted at 2014-11-03 13:09:48 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:48.811354 2173 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master uuid: "2facf84ee8b640bb8e539aa686f8039c" format_stamp: "Formatted at 2014-11-03 13:09:48 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:48.814362 2173 master_main.cc:30] Starting master server... I1103 05:09:48.816601 2173 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:48.816634 2173 webserver.cc:117] Document root disabled I1103 05:09:48.817095 2173 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:38835 I1103 05:09:48.817287 2173 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb I1103 05:09:48.825800 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:57700 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:48.845294 2267 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:48.860849 2267 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/instance: uuid: "655a3e4e2b48486383697db1fb272835" format_stamp: "Formatted at 2014-11-03 13:09:48 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:48.860976 2267 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 uuid: "655a3e4e2b48486383697db1fb272835" format_stamp: "Formatted at 2014-11-03 13:09:48 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:48.863862 2267 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:48.867501 2267 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:48.867532 2267 webserver.cc:117] Document root disabled I1103 05:09:48.868011 2267 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:39734 I1103 05:09:48.868551 2267 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb I1103 05:09:48.868850 2267 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:48.869127 2267 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:48.870707 2380 heartbeater.cc:227] Registering TS with master... I1103 05:09:48.870729 2380 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:48.878053 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:57700 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:48.879509 2248 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: Bootstrap starting. I1103 05:09:48.879912 2248 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one. I1103 05:09:48.880204 2248 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:09:48.880363 2412 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:09:48.880602 2248 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: No bootstrap required, opened a new log I1103 05:09:48.881769 2414 tablet_peer.cc:187] Configuration changed for tablet: 11111111111111111111111111111111 in peer: 2facf84ee8b640bb8e539aa686f8039c Changed from: seqno: 0 local: true peers { permanent_uuid: "2facf84ee8b640bb8e539aa686f8039c" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "2facf84ee8b640bb8e539aa686f8039c" role: LEADER } I1103 05:09:48.881824 2414 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "2facf84ee8b640bb8e539aa686f8039c" role: LEADER } I1103 05:09:48.899657 2410 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:48.947968 2410 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/instance: uuid: "07cb960abedc43a7a8bea75e1b465197" format_stamp: "Formatted at 2014-11-03 13:09:48 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:48.948149 2410 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 uuid: "07cb960abedc43a7a8bea75e1b465197" format_stamp: "Formatted at 2014-11-03 13:09:48 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:48.953349 2410 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:48.956737 2410 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:48.956764 2410 webserver.cc:117] Document root disabled I1103 05:09:48.957267 2410 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:33573 I1103 05:09:48.957449 2410 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb I1103 05:09:48.957564 2410 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:48.957665 2410 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:48.958699 2633 heartbeater.cc:227] Registering TS with master... I1103 05:09:48.958721 2633 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:48.960701 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:57700 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:48.982007 2636 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:49.026234 2636 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/instance: uuid: "6c5033bd4ec64c09b084bd11860c4d04" format_stamp: "Formatted at 2014-11-03 13:09:48 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:49.031427 2636 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 uuid: "6c5033bd4ec64c09b084bd11860c4d04" format_stamp: "Formatted at 2014-11-03 13:09:48 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:49.031553 2248 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: Bootstrap starting. I1103 05:09:49.031981 2248 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I1103 05:09:49.032569 2248 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I1103 05:09:49.036586 2636 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:49.038753 2248 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. W1103 05:09:49.038884 2229 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:09:49.038991 2227 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. I1103 05:09:49.039219 2763 tablet_peer.cc:187] Configuration changed for tablet: 00000000000000000000000000000000 in peer: 2facf84ee8b640bb8e539aa686f8039c Changed from: seqno: 0 local: true peers { permanent_uuid: "2facf84ee8b640bb8e539aa686f8039c" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "2facf84ee8b640bb8e539aa686f8039c" role: LEADER } I1103 05:09:49.039247 2763 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "2facf84ee8b640bb8e539aa686f8039c" role: LEADER } I1103 05:09:49.039254 2248 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:09:49.039352 2173 master_main.cc:33] Master server successfully started. I1103 05:09:49.042474 2636 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:49.042505 2636 webserver.cc:117] Document root disabled I1103 05:09:49.043078 2636 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:33461 I1103 05:09:49.043305 2636 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertAndMutateThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb I1103 05:09:49.043431 2636 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:49.043534 2636 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:49.045081 2870 heartbeater.cc:227] Registering TS with master... I1103 05:09:49.045102 2870 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:49.045430 2232 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "6c5033bd4ec64c09b084bd11860c4d04" instance_seqno: 1415020189042344 } with Master I1103 05:09:50.039357 2227 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" instance_seqno: 1415020188956585 } as {real_user=jenkins, eff_user=} at 127.0.0.1:33627; Asking this server to re-register. I1103 05:09:50.039489 2231 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "655a3e4e2b48486383697db1fb272835" instance_seqno: 1415020188867359 } as {real_user=jenkins, eff_user=} at 127.0.0.1:33618; Asking this server to re-register. I1103 05:09:50.039602 2633 heartbeater.cc:227] Registering TS with master... I1103 05:09:50.039633 2633 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:50.039767 2230 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" instance_seqno: 1415020188956585 } with Master I1103 05:09:50.039847 2380 heartbeater.cc:227] Registering TS with master... I1103 05:09:50.039872 2380 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:50.039974 2232 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "655a3e4e2b48486383697db1fb272835" instance_seqno: 1415020188867359 } with Master I1103 05:09:50.040458 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:09:50.040776 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:09:50.043416 2226 catalog_manager.cc:437] CreateTable from {real_user=jenkins, eff_user=} at 127.0.0.1:33685: name: "TestTable" schema { columns { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I1103 05:09:50.046551 2845 tablet_service.cc:264] Processing CreateTablet for tablet 607a992db198406ea346bc4267b364fa (table=TestTable [id=90302ae91ab64e70b9890906827d02b3]), range=["", ""] I1103 05:09:50.046871 2338 tablet_service.cc:264] Processing CreateTablet for tablet 607a992db198406ea346bc4267b364fa (table=TestTable [id=90302ae91ab64e70b9890906827d02b3]), range=["", ""] I1103 05:09:50.047216 2610 tablet_service.cc:264] Processing CreateTablet for tablet 607a992db198406ea346bc4267b364fa (table=TestTable [id=90302ae91ab64e70b9890906827d02b3]), range=["", ""] I1103 05:09:50.473304 2338 ts_tablet_manager.cc:464] Registered tablet 607a992db198406ea346bc4267b364fa I1103 05:09:50.473466 2610 ts_tablet_manager.cc:464] Registered tablet 607a992db198406ea346bc4267b364fa I1103 05:09:50.473572 3575 ts_tablet_manager.cc:325] Bootstrapping tablet: 607a992db198406ea346bc4267b364fa I1103 05:09:50.473582 2845 ts_tablet_manager.cc:464] Registered tablet 607a992db198406ea346bc4267b364fa I1103 05:09:50.473677 3574 ts_tablet_manager.cc:325] Bootstrapping tablet: 607a992db198406ea346bc4267b364fa I1103 05:09:50.473765 3575 tablet_bootstrap.cc:234] Tablet 607a992db198406ea346bc4267b364fa: Bootstrap starting. I1103 05:09:50.473816 3576 ts_tablet_manager.cc:325] Bootstrapping tablet: 607a992db198406ea346bc4267b364fa I1103 05:09:50.473878 3574 tablet_bootstrap.cc:234] Tablet 607a992db198406ea346bc4267b364fa: Bootstrap starting. I1103 05:09:50.473964 3576 tablet_bootstrap.cc:234] Tablet 607a992db198406ea346bc4267b364fa: Bootstrap starting. I1103 05:09:50.474373 3575 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 607a992db198406ea346bc4267b364fa creating new one. I1103 05:09:50.474408 3574 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 607a992db198406ea346bc4267b364fa creating new one. I1103 05:09:50.474493 3576 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 607a992db198406ea346bc4267b364fa creating new one. I1103 05:09:50.474717 3575 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:09:50.474756 3574 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:09:50.474850 3576 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:09:50.474973 3583 env_posix.cc:511] The filesystem does not support fallocate(). W1103 05:09:50.475042 3585 env_posix.cc:511] The filesystem does not support fallocate(). W1103 05:09:50.475189 3582 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:09:50.475206 3574 tablet_bootstrap.cc:234] Tablet 607a992db198406ea346bc4267b364fa: No bootstrap required, opened a new log I1103 05:09:50.475263 3574 ts_tablet_manager.cc:330] Time spent tablet 607a992db198406ea346bc4267b364fa bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:09:50.475587 3575 tablet_bootstrap.cc:234] Tablet 607a992db198406ea346bc4267b364fa: No bootstrap required, opened a new log I1103 05:09:50.475628 3574 raft_consensus.cc:102] T 607a992db198406ea346bc4267b364fa P 655a3e4e2b48486383697db1fb272835 [CANDIDATE]: Replica starting. Triggering 0 pending transactions. I1103 05:09:50.475636 3575 ts_tablet_manager.cc:330] Time spent tablet 607a992db198406ea346bc4267b364fa bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:09:50.475750 3576 tablet_bootstrap.cc:234] Tablet 607a992db198406ea346bc4267b364fa: No bootstrap required, opened a new log I1103 05:09:50.475811 3576 ts_tablet_manager.cc:330] Time spent tablet 607a992db198406ea346bc4267b364fa bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:09:50.475958 3574 raft_consensus.cc:235] T 607a992db198406ea346bc4267b364fa P 655a3e4e2b48486383697db1fb272835 [LEADER]: Becoming Leader I1103 05:09:50.476035 3575 raft_consensus.cc:102] T 607a992db198406ea346bc4267b364fa P 07cb960abedc43a7a8bea75e1b465197 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:09:50.476191 3575 raft_consensus.cc:302] T 607a992db198406ea346bc4267b364fa P 07cb960abedc43a7a8bea75e1b465197 [FOLLOWER]: Becoming Follower/Learner I1103 05:09:50.476269 3575 ts_tablet_manager.cc:350] Time spent starting tablet 607a992db198406ea346bc4267b364fa: real 0.001s user 0.001s sys 0.000s I1103 05:09:50.476265 3576 raft_consensus.cc:102] T 607a992db198406ea346bc4267b364fa P 6c5033bd4ec64c09b084bd11860c4d04 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:09:50.476377 3576 raft_consensus.cc:302] T 607a992db198406ea346bc4267b364fa P 6c5033bd4ec64c09b084bd11860c4d04 [FOLLOWER]: Becoming Follower/Learner I1103 05:09:50.476433 3576 ts_tablet_manager.cc:350] Time spent starting tablet 607a992db198406ea346bc4267b364fa: real 0.001s user 0.000s sys 0.000s I1103 05:09:50.477463 3574 ts_tablet_manager.cc:350] Time spent starting tablet 607a992db198406ea346bc4267b364fa: real 0.002s user 0.001s sys 0.001s W1103 05:09:50.976481 3592 raft_consensus_state.cc:463] T 607a992db198406ea346bc4267b364fa P 655a3e4e2b48486383697db1fb272835 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 W1103 05:09:50.976877 3586 raft_consensus_state.cc:463] T 607a992db198406ea346bc4267b364fa P 655a3e4e2b48486383697db1fb272835 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:09:50.978188 2620 raft_consensus.cc:608] T 607a992db198406ea346bc4267b364fa P 07cb960abedc43a7a8bea75e1b465197 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:09:50.978322 2324 consensus_queue.cc:451] Connected to peer: 07cb960abedc43a7a8bea75e1b465197 whose last received watermark is: term: 0 index: 0 I1103 05:09:50.979446 2324 raft_consensus_state.cc:456] T 607a992db198406ea346bc4267b364fa P 655a3e4e2b48486383697db1fb272835 [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I1103 05:09:50.979485 2858 raft_consensus.cc:608] T 607a992db198406ea346bc4267b364fa P 6c5033bd4ec64c09b084bd11860c4d04 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:09:50.979562 3586 tablet_peer.cc:187] Configuration changed for tablet: 607a992db198406ea346bc4267b364fa in peer: 655a3e4e2b48486383697db1fb272835 Changed from: seqno: -1 local: false peers { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40929 } } peers { permanent_uuid: "6c5033bd4ec64c09b084bd11860c4d04" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58567 } } peers { permanent_uuid: "655a3e4e2b48486383697db1fb272835" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 36186 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40929 } } peers { permanent_uuid: "6c5033bd4ec64c09b084bd11860c4d04" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58567 } } peers { permanent_uuid: "655a3e4e2b48486383697db1fb272835" role: LEADER last_known_addr { host: "127.0.0.1" port: 36186 } } I1103 05:09:50.979637 2325 consensus_queue.cc:451] Connected to peer: 6c5033bd4ec64c09b084bd11860c4d04 whose last received watermark is: term: 0 index: 0 I1103 05:09:50.980212 3589 tablet_peer.cc:187] Configuration changed for tablet: 607a992db198406ea346bc4267b364fa in peer: 6c5033bd4ec64c09b084bd11860c4d04 Changed from: seqno: -1 local: false peers { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40929 } } peers { permanent_uuid: "6c5033bd4ec64c09b084bd11860c4d04" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58567 } } peers { permanent_uuid: "655a3e4e2b48486383697db1fb272835" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 36186 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40929 } } peers { permanent_uuid: "6c5033bd4ec64c09b084bd11860c4d04" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58567 } } peers { permanent_uuid: "655a3e4e2b48486383697db1fb272835" role: LEADER last_known_addr { host: "127.0.0.1" port: 36186 } } I1103 05:09:51.040428 2233 catalog_manager.cc:1127] Tablet: 607a992db198406ea346bc4267b364fa reported quorum change. New quorum: seqno: 0 local: false peers { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40929 } } peers { permanent_uuid: "6c5033bd4ec64c09b084bd11860c4d04" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58567 } } peers { permanent_uuid: "655a3e4e2b48486383697db1fb272835" role: LEADER last_known_addr { host: "127.0.0.1" port: 36186 } } I1103 05:09:51.107273 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master W1103 05:09:51.107697 2172 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:57700, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:09:51.113512 3587 tablet_peer.cc:187] Configuration changed for tablet: 607a992db198406ea346bc4267b364fa in peer: 07cb960abedc43a7a8bea75e1b465197 Changed from: seqno: -1 local: false peers { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40929 } } peers { permanent_uuid: "6c5033bd4ec64c09b084bd11860c4d04" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58567 } } peers { permanent_uuid: "655a3e4e2b48486383697db1fb272835" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 36186 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "07cb960abedc43a7a8bea75e1b465197" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40929 } } peers { permanent_uuid: "6c5033bd4ec64c09b084bd11860c4d04" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58567 } } peers { permanent_uuid: "655a3e4e2b48486383697db1fb272835" role: LEADER last_known_addr { host: "127.0.0.1" port: 36186 } } W1103 05:09:51.268633 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.77-1.77] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:51.284483 3587 log.cc:507] Time spent Append to log took a long time: real 0.066s user 0.001s sys 0.000s W1103 05:09:51.284536 2628 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 78) took 66ms (client timeout 50). W1103 05:09:51.284636 2628 inbound_call.cc:140] Trace: 1103 05:09:51.218298 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:51.218317 (+ 19us) service_pool.cc:152] Handling call 1103 05:09:51.218329 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:51.218331 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:51.218354 (+ 23us) log.cc:406] Serialized 485 byte log entry 1103 05:09:51.218360 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 76 1103 05:09:51.218368 (+ 8us) raft_consensus.cc:682] Updating last received op as term: 1 index: 77 1103 05:09:51.218369 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:51.284518 (+ 66149us) raft_consensus.cc:697] finished 1103 05:09:51.284520 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:51.284522 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:51.284527 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:51.218391 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:51.218396 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:51.218397 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:51.218398 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:51.218407 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:51.218415 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:51.218416 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:51.218418 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:51.218421 (+ 3us) write_transaction.cc:95] Timestamp: L: 151 W1103 05:09:51.289425 3586 log.cc:507] Time spent Append to log took a long time: real 0.071s user 0.000s sys 0.000s W1103 05:09:51.351053 2861 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 94) took 48ms (client timeout 50). W1103 05:09:51.351171 2861 inbound_call.cc:140] Trace: 1103 05:09:51.302830 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:51.302861 (+ 31us) service_pool.cc:152] Handling call 1103 05:09:51.302873 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:51.302874 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:51.302898 (+ 24us) log.cc:406] Serialized 485 byte log entry 1103 05:09:51.302905 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 92 1103 05:09:51.302915 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 93 1103 05:09:51.302917 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:51.351029 (+ 48112us) raft_consensus.cc:697] finished 1103 05:09:51.351031 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:51.351033 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:51.351038 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:51.302900 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:51.302904 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:51.302905 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:51.302906 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:51.302912 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:51.302918 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:51.302919 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:51.302920 (+ 1us) write_transaction.cc:93] Start() 1103 05:09:51.302923 (+ 3us) write_transaction.cc:95] Timestamp: L: 183 W1103 05:09:51.352762 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.93-1.93] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:51.353129 3587 log.cc:507] Time spent Append to log took a long time: real 0.050s user 0.000s sys 0.000s W1103 05:09:51.353178 2623 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 94) took 50ms (client timeout 50). W1103 05:09:51.353217 2623 inbound_call.cc:140] Trace: 1103 05:09:51.302625 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:51.302641 (+ 16us) service_pool.cc:152] Handling call 1103 05:09:51.302650 (+ 9us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:51.302652 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:51.302676 (+ 24us) log.cc:406] Serialized 485 byte log entry 1103 05:09:51.302685 (+ 9us) raft_consensus.cc:669] Marking committed up to term: 1 index: 92 1103 05:09:51.302696 (+ 11us) raft_consensus.cc:682] Updating last received op as term: 1 index: 93 1103 05:09:51.302698 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:51.353161 (+ 50463us) raft_consensus.cc:697] finished 1103 05:09:51.353163 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:51.353165 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:51.353170 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:51.302936 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:51.302941 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:51.302942 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:51.302943 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:51.302949 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:51.302956 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:51.302956 (+ 0us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:51.302958 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:51.302961 (+ 3us) write_transaction.cc:95] Timestamp: L: 183 W1103 05:09:51.355646 3586 log.cc:507] Time spent Append to log took a long time: real 0.053s user 0.000s sys 0.000s W1103 05:09:51.485354 2628 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 128) took 50ms (client timeout 50). W1103 05:09:51.485402 2628 inbound_call.cc:140] Trace: 1103 05:09:51.435220 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:51.435241 (+ 21us) service_pool.cc:152] Handling call 1103 05:09:51.435253 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:51.435254 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:51.435279 (+ 25us) log.cc:406] Serialized 495 byte log entry 1103 05:09:51.435286 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 126 1103 05:09:51.435296 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 127 1103 05:09:51.435298 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:51.485331 (+ 50033us) raft_consensus.cc:697] finished 1103 05:09:51.485333 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:51.485335 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:51.485340 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:51.435282 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:51.435287 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:51.435288 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:51.435289 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:51.435295 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:51.435304 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:51.435305 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:51.435307 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:51.435310 (+ 3us) write_transaction.cc:95] Timestamp: L: 251 W1103 05:09:51.485481 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.127-1.127] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:51.485602 2325 consensus_peers.cc:368] Couldn't send request to peer 6c5033bd4ec64c09b084bd11860c4d04 for tablet 607a992db198406ea346bc4267b364fa ops [1.127-1.127] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:51.489210 3589 log.cc:507] Time spent Append to log took a long time: real 0.054s user 0.000s sys 0.000s W1103 05:09:51.489261 2865 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 128) took 54ms (client timeout 50). W1103 05:09:51.489305 2865 inbound_call.cc:140] Trace: 1103 05:09:51.435166 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:51.435185 (+ 19us) service_pool.cc:152] Handling call 1103 05:09:51.435198 (+ 13us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:51.435200 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:51.435226 (+ 26us) log.cc:406] Serialized 495 byte log entry 1103 05:09:51.435233 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 126 1103 05:09:51.435243 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 127 1103 05:09:51.435245 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:51.489244 (+ 53999us) raft_consensus.cc:697] finished 1103 05:09:51.489246 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:51.489247 (+ 1us) raft_consensus.cc:443] Updating watermarks 1103 05:09:51.489252 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:51.435256 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:51.435261 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:51.435262 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:51.435264 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:51.435272 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:51.435279 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:51.435280 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:51.435281 (+ 1us) write_transaction.cc:93] Start() 1103 05:09:51.435284 (+ 3us) write_transaction.cc:95] Timestamp: L: 251 W1103 05:09:51.512132 3586 log.cc:507] Time spent Append to log took a long time: real 0.077s user 0.000s sys 0.000s W1103 05:09:52.222450 2867 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 220) took 38ms (client timeout 50). W1103 05:09:52.222503 2867 inbound_call.cc:140] Trace: 1103 05:09:52.184177 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.184217 (+ 40us) service_pool.cc:152] Handling call 1103 05:09:52.184229 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.184231 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.184259 (+ 28us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.184266 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 217 1103 05:09:52.184278 (+ 12us) raft_consensus.cc:682] Updating last received op as term: 1 index: 218 1103 05:09:52.184280 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:52.222430 (+ 38150us) raft_consensus.cc:697] finished 1103 05:09:52.222431 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:52.222433 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:52.222438 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.184258 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.184263 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.184265 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.184266 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.184272 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.184281 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.184282 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.184284 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:52.184287 (+ 3us) write_transaction.cc:95] Timestamp: L: 433 W1103 05:09:52.298565 2325 consensus_peers.cc:368] Couldn't send request to peer 6c5033bd4ec64c09b084bd11860c4d04 for tablet 607a992db198406ea346bc4267b364fa ops [1.226-1.226] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:52.298940 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.226-1.226] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:52.318277 3587 log.cc:507] Time spent Append to log took a long time: real 0.070s user 0.000s sys 0.000s W1103 05:09:52.318370 2628 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 228) took 69ms (client timeout 50). W1103 05:09:52.318421 2628 inbound_call.cc:140] Trace: 1103 05:09:52.248479 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.248525 (+ 46us) service_pool.cc:152] Handling call 1103 05:09:52.248537 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.248539 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.248560 (+ 21us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.248566 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 225 1103 05:09:52.248574 (+ 8us) raft_consensus.cc:682] Updating last received op as term: 1 index: 226 1103 05:09:52.248575 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:52.318350 (+ 69775us) raft_consensus.cc:697] finished 1103 05:09:52.318352 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:52.318355 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:09:52.318360 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.248615 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.248620 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.248622 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.248623 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.248633 (+ 10us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.248642 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.248643 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.248645 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:52.248648 (+ 3us) write_transaction.cc:95] Timestamp: L: 449 W1103 05:09:52.322818 3589 log.cc:507] Time spent Append to log took a long time: real 0.075s user 0.000s sys 0.000s W1103 05:09:52.322897 2865 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 228) took 74ms (client timeout 50). W1103 05:09:52.322954 2865 inbound_call.cc:140] Trace: 1103 05:09:52.248026 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.248045 (+ 19us) service_pool.cc:152] Handling call 1103 05:09:52.248056 (+ 11us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.248058 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.248081 (+ 23us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.248088 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 225 1103 05:09:52.248098 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 226 1103 05:09:52.248100 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:52.322878 (+ 74778us) raft_consensus.cc:697] finished 1103 05:09:52.322880 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:52.322882 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:52.322887 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.248376 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.248382 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.248383 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.248384 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.248392 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.248403 (+ 11us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.248404 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.248406 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:52.248411 (+ 5us) write_transaction.cc:95] Timestamp: L: 449 W1103 05:09:52.793540 2863 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 236) took 39ms (client timeout 50). W1103 05:09:52.793547 2625 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 236) took 39ms (client timeout 50). W1103 05:09:52.793615 2863 inbound_call.cc:140] Trace: 1103 05:09:52.754217 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.754242 (+ 25us) service_pool.cc:152] Handling call 1103 05:09:52.754255 (+ 13us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.754258 (+ 3us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.754302 (+ 44us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.754310 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 232 1103 05:09:52.754328 (+ 18us) raft_consensus.cc:682] Updating last received op as term: 1 index: 233 1103 05:09:52.754329 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:52.793491 (+ 39162us) raft_consensus.cc:697] finished 1103 05:09:52.793493 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:52.793496 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:09:52.793503 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.754302 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.754308 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.754309 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.754310 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.754318 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.754327 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.754329 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.754330 (+ 1us) write_transaction.cc:93] Start() 1103 05:09:52.754333 (+ 3us) write_transaction.cc:95] Timestamp: L: 463 W1103 05:09:52.793620 2625 inbound_call.cc:140] Trace: 1103 05:09:52.753926 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.753947 (+ 21us) service_pool.cc:152] Handling call 1103 05:09:52.753963 (+ 16us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.753965 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.753995 (+ 30us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.754002 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 232 1103 05:09:52.754012 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 233 1103 05:09:52.754014 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:52.793508 (+ 39494us) raft_consensus.cc:697] finished 1103 05:09:52.793511 (+ 3us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:52.793514 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:09:52.793521 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.754001 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.754008 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.754009 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.754010 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.754019 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.754028 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.754029 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.754031 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:52.754035 (+ 4us) write_transaction.cc:95] Timestamp: L: 463 W1103 05:09:52.832998 2864 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 237) took 38ms (client timeout 50). W1103 05:09:52.833086 2864 inbound_call.cc:140] Trace: 1103 05:09:52.794802 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.794828 (+ 26us) service_pool.cc:152] Handling call 1103 05:09:52.794843 (+ 15us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.794844 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.794893 (+ 49us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.794902 (+ 9us) raft_consensus.cc:669] Marking committed up to term: 1 index: 233 1103 05:09:52.794919 (+ 17us) raft_consensus.cc:682] Updating last received op as term: 1 index: 234 1103 05:09:52.794921 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:52.832949 (+ 38028us) raft_consensus.cc:697] finished 1103 05:09:52.832951 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:52.832954 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:09:52.832961 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.794904 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.794912 (+ 8us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.794913 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.794914 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.794926 (+ 12us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.794941 (+ 15us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.794942 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.794944 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:52.794948 (+ 4us) write_transaction.cc:95] Timestamp: L: 465 W1103 05:09:52.882485 2866 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 249) took 39ms (client timeout 50). W1103 05:09:52.882536 2630 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 250) took 39ms (client timeout 50). W1103 05:09:52.882562 2866 inbound_call.cc:140] Trace: 1103 05:09:52.842533 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.842584 (+ 51us) service_pool.cc:152] Handling call 1103 05:09:52.842596 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.842598 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.842629 (+ 31us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.842636 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 246 1103 05:09:52.842646 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 247 1103 05:09:52.842648 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:52.882455 (+ 39807us) raft_consensus.cc:697] finished 1103 05:09:52.882458 (+ 3us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:52.882460 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:52.882468 (+ 8us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.842631 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.842635 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.842637 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.842637 (+ 0us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.842644 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.842652 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.842653 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.842654 (+ 1us) write_transaction.cc:93] Start() 1103 05:09:52.842657 (+ 3us) write_transaction.cc:95] Timestamp: L: 491 W1103 05:09:52.882586 2630 inbound_call.cc:140] Trace: 1103 05:09:52.842575 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.842596 (+ 21us) service_pool.cc:152] Handling call 1103 05:09:52.842608 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.842610 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.842639 (+ 29us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.842646 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 246 1103 05:09:52.842656 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 247 1103 05:09:52.842657 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:52.882510 (+ 39853us) raft_consensus.cc:697] finished 1103 05:09:52.882512 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:52.882515 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:09:52.882521 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.842670 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.842676 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.842677 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.842679 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.842686 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.842696 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.842697 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.842699 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:52.842702 (+ 3us) write_transaction.cc:95] Timestamp: L: 491 W1103 05:09:53.009464 2325 consensus_peers.cc:368] Couldn't send request to peer 6c5033bd4ec64c09b084bd11860c4d04 for tablet 607a992db198406ea346bc4267b364fa ops [1.268-1.268] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:53.009568 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.268-1.268] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:53.038578 3587 log.cc:507] Time spent Append to log took a long time: real 0.079s user 0.000s sys 0.000s W1103 05:09:53.038641 2620 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 271) took 79ms (client timeout 50). W1103 05:09:53.038691 2620 inbound_call.cc:140] Trace: 1103 05:09:52.959379 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.959399 (+ 20us) service_pool.cc:152] Handling call 1103 05:09:52.959411 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.959413 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.959441 (+ 28us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.959446 (+ 5us) raft_consensus.cc:669] Marking committed up to term: 1 index: 267 1103 05:09:52.959455 (+ 9us) raft_consensus.cc:682] Updating last received op as term: 1 index: 268 1103 05:09:52.959456 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:53.038619 (+ 79163us) raft_consensus.cc:697] finished 1103 05:09:53.038621 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:53.038623 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:53.038630 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.959457 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.959463 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.959464 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.959465 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.959472 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.959480 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.959481 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.959483 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:52.959487 (+ 4us) write_transaction.cc:95] Timestamp: L: 533 W1103 05:09:53.038702 3589 log.cc:507] Time spent Append to log took a long time: real 0.079s user 0.000s sys 0.000s W1103 05:09:53.038800 2867 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 270) took 79ms (client timeout 50). W1103 05:09:53.038848 2867 inbound_call.cc:140] Trace: 1103 05:09:52.959093 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:52.959265 (+ 172us) service_pool.cc:152] Handling call 1103 05:09:52.959279 (+ 14us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:52.959280 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:52.959311 (+ 31us) log.cc:406] Serialized 496 byte log entry 1103 05:09:52.959319 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 267 1103 05:09:52.959329 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 268 1103 05:09:52.959331 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:53.038774 (+ 79443us) raft_consensus.cc:697] finished 1103 05:09:53.038776 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:53.038779 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:09:53.038786 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:52.959342 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:52.959348 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:52.959349 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:52.959350 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:52.959359 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:52.959368 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:52.959369 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:52.959371 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:52.959375 (+ 4us) write_transaction.cc:95] Timestamp: L: 533 W1103 05:09:53.533627 2625 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 306) took 43ms (client timeout 50). W1103 05:09:53.533675 2625 inbound_call.cc:140] Trace: 1103 05:09:53.490606 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:53.490628 (+ 22us) service_pool.cc:152] Handling call 1103 05:09:53.490640 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:53.490641 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:53.490663 (+ 22us) log.cc:406] Serialized 496 byte log entry 1103 05:09:53.490668 (+ 5us) raft_consensus.cc:669] Marking committed up to term: 1 index: 301 1103 05:09:53.490676 (+ 8us) raft_consensus.cc:682] Updating last received op as term: 1 index: 302 1103 05:09:53.490677 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:53.533609 (+ 42932us) raft_consensus.cc:697] finished 1103 05:09:53.533611 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:53.533613 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:53.533617 (+ 4us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:53.490672 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:53.490677 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:53.490678 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:53.490680 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:53.490687 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:53.490696 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:53.490697 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:53.490699 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:53.490702 (+ 3us) write_transaction.cc:95] Timestamp: L: 601 W1103 05:09:53.538163 2862 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 305) took 47ms (client timeout 50). W1103 05:09:53.538224 2862 inbound_call.cc:140] Trace: 1103 05:09:53.490575 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:53.490598 (+ 23us) service_pool.cc:152] Handling call 1103 05:09:53.490607 (+ 9us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:53.490609 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:53.490631 (+ 22us) log.cc:406] Serialized 496 byte log entry 1103 05:09:53.490638 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 301 1103 05:09:53.490649 (+ 11us) raft_consensus.cc:682] Updating last received op as term: 1 index: 302 1103 05:09:53.490650 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:53.538144 (+ 47494us) raft_consensus.cc:697] finished 1103 05:09:53.538146 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:53.538148 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:53.538152 (+ 4us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:53.490649 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:53.490653 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:53.490654 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:53.490655 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:53.490662 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:53.490672 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:53.490673 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:53.490675 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:53.490678 (+ 3us) write_transaction.cc:95] Timestamp: L: 601 W1103 05:09:53.590756 2629 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 319) took 43ms (client timeout 50). W1103 05:09:53.590803 2629 inbound_call.cc:140] Trace: 1103 05:09:53.547266 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:53.547297 (+ 31us) service_pool.cc:152] Handling call 1103 05:09:53.547309 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:53.547311 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:53.547346 (+ 35us) log.cc:406] Serialized 496 byte log entry 1103 05:09:53.547354 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 314 1103 05:09:53.547367 (+ 13us) raft_consensus.cc:682] Updating last received op as term: 1 index: 315 1103 05:09:53.547369 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:53.590738 (+ 43369us) raft_consensus.cc:697] finished 1103 05:09:53.590740 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:53.590742 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:53.590746 (+ 4us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:53.547348 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:53.547353 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:53.547354 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:53.547355 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:53.547439 (+ 84us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:53.547446 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:53.547447 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:53.547449 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:53.547452 (+ 3us) write_transaction.cc:95] Timestamp: L: 627 W1103 05:09:53.594434 2860 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 313) took 47ms (client timeout 50). W1103 05:09:53.594498 2860 inbound_call.cc:140] Trace: 1103 05:09:53.546689 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:53.546713 (+ 24us) service_pool.cc:152] Handling call 1103 05:09:53.546725 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:53.546726 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:53.546757 (+ 31us) log.cc:406] Serialized 496 byte log entry 1103 05:09:53.546765 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 314 1103 05:09:53.546776 (+ 11us) raft_consensus.cc:682] Updating last received op as term: 1 index: 315 1103 05:09:53.546777 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:53.594416 (+ 47639us) raft_consensus.cc:697] finished 1103 05:09:53.594418 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:53.594420 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:53.594424 (+ 4us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:53.546759 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:53.546764 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:53.546765 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:53.546766 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:53.546773 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:53.546782 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:53.546783 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:53.546786 (+ 3us) write_transaction.cc:93] Start() 1103 05:09:53.546789 (+ 3us) write_transaction.cc:95] Timestamp: L: 627 W1103 05:09:53.765442 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.348-1.348] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:53.765554 2325 consensus_peers.cc:368] Couldn't send request to peer 6c5033bd4ec64c09b084bd11860c4d04 for tablet 607a992db198406ea346bc4267b364fa ops [1.348-1.348] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:53.767199 3589 log.cc:507] Time spent Append to log took a long time: real 0.052s user 0.000s sys 0.000s W1103 05:09:53.767253 2863 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 346) took 52ms (client timeout 50). W1103 05:09:53.767297 2863 inbound_call.cc:140] Trace: 1103 05:09:53.715234 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:53.715252 (+ 18us) service_pool.cc:152] Handling call 1103 05:09:53.715261 (+ 9us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:53.715263 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:53.715285 (+ 22us) log.cc:406] Serialized 496 byte log entry 1103 05:09:53.715290 (+ 5us) raft_consensus.cc:669] Marking committed up to term: 1 index: 347 1103 05:09:53.715297 (+ 7us) raft_consensus.cc:682] Updating last received op as term: 1 index: 348 1103 05:09:53.715299 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:53.767235 (+ 51936us) raft_consensus.cc:697] finished 1103 05:09:53.767237 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:53.767239 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:53.767244 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:53.715305 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:53.715310 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:53.715311 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:53.715313 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:53.715321 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:53.715330 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:53.715331 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:53.715333 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:53.715336 (+ 3us) write_transaction.cc:95] Timestamp: L: 693 W1103 05:09:53.767681 3587 log.cc:507] Time spent Append to log took a long time: real 0.052s user 0.000s sys 0.000s W1103 05:09:53.767732 2621 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 342) took 52ms (client timeout 50). W1103 05:09:53.767776 2621 inbound_call.cc:140] Trace: 1103 05:09:53.715110 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:53.715131 (+ 21us) service_pool.cc:152] Handling call 1103 05:09:53.715142 (+ 11us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:53.715144 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:53.715170 (+ 26us) log.cc:406] Serialized 496 byte log entry 1103 05:09:53.715177 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 347 1103 05:09:53.715188 (+ 11us) raft_consensus.cc:682] Updating last received op as term: 1 index: 348 1103 05:09:53.715190 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:53.767716 (+ 52526us) raft_consensus.cc:697] finished 1103 05:09:53.767718 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:53.767719 (+ 1us) raft_consensus.cc:443] Updating watermarks 1103 05:09:53.767724 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:53.715172 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:53.715177 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:53.715178 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:53.715179 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:53.715186 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:53.715196 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:53.715197 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:53.715199 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:53.715202 (+ 3us) write_transaction.cc:95] Timestamp: L: 692 W1103 05:09:53.773157 3586 log.cc:507] Time spent Append to log took a long time: real 0.058s user 0.000s sys 0.000s W1103 05:09:54.323559 2325 consensus_peers.cc:368] Couldn't send request to peer 6c5033bd4ec64c09b084bd11860c4d04 for tablet 607a992db198406ea346bc4267b364fa ops [1.364-1.364] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:54.323556 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.364-1.364] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:54.324592 3587 log.cc:507] Time spent Append to log took a long time: real 0.051s user 0.000s sys 0.000s W1103 05:09:54.324645 2629 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 359) took 51ms (client timeout 50). W1103 05:09:54.324687 2629 inbound_call.cc:140] Trace: 1103 05:09:54.273347 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:54.273362 (+ 15us) service_pool.cc:152] Handling call 1103 05:09:54.273370 (+ 8us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:54.273372 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:54.273396 (+ 24us) log.cc:406] Serialized 496 byte log entry 1103 05:09:54.273401 (+ 5us) raft_consensus.cc:669] Marking committed up to term: 1 index: 363 1103 05:09:54.273410 (+ 9us) raft_consensus.cc:682] Updating last received op as term: 1 index: 364 1103 05:09:54.273412 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:54.324628 (+ 51216us) raft_consensus.cc:697] finished 1103 05:09:54.324629 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:54.324631 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:54.324636 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:54.273405 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:54.273410 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:54.273412 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:09:54.273413 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:54.273423 (+ 10us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:54.273432 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:54.273434 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:54.273436 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:54.273439 (+ 3us) write_transaction.cc:95] Timestamp: L: 724 W1103 05:09:54.324723 3589 log.cc:507] Time spent Append to log took a long time: real 0.051s user 0.000s sys 0.000s W1103 05:09:54.324782 2860 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 363) took 51ms (client timeout 50). W1103 05:09:54.324839 2860 inbound_call.cc:140] Trace: 1103 05:09:54.273348 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:54.273366 (+ 18us) service_pool.cc:152] Handling call 1103 05:09:54.273377 (+ 11us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:54.273378 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:54.273409 (+ 31us) log.cc:406] Serialized 496 byte log entry 1103 05:09:54.273416 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 363 1103 05:09:54.273429 (+ 13us) raft_consensus.cc:682] Updating last received op as term: 1 index: 364 1103 05:09:54.273431 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:54.324762 (+ 51331us) raft_consensus.cc:697] finished 1103 05:09:54.324764 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:54.324766 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:54.324772 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:54.273442 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:54.273447 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:54.273448 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:54.273449 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:54.273458 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:54.273467 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:54.273468 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:54.273470 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:54.273473 (+ 3us) write_transaction.cc:95] Timestamp: L: 725 W1103 05:09:54.328482 3586 log.cc:507] Time spent Append to log took a long time: real 0.055s user 0.000s sys 0.000s W1103 05:09:54.843746 2864 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 377) took 39ms (client timeout 50). W1103 05:09:54.843811 2864 inbound_call.cc:140] Trace: 1103 05:09:54.803975 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:54.803998 (+ 23us) service_pool.cc:152] Handling call 1103 05:09:54.804009 (+ 11us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:54.804010 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:54.804033 (+ 23us) log.cc:406] Serialized 496 byte log entry 1103 05:09:54.804038 (+ 5us) raft_consensus.cc:669] Marking committed up to term: 1 index: 376 1103 05:09:54.804045 (+ 7us) raft_consensus.cc:682] Updating last received op as term: 1 index: 377 1103 05:09:54.804047 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:54.843723 (+ 39676us) raft_consensus.cc:697] finished 1103 05:09:54.843725 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:54.843727 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:54.843733 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:54.804071 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:54.804076 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:54.804077 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:54.804078 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:54.804172 (+ 94us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:54.804180 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:54.804181 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:54.804182 (+ 1us) write_transaction.cc:93] Start() 1103 05:09:54.804185 (+ 3us) write_transaction.cc:95] Timestamp: L: 751 W1103 05:09:54.847300 2622 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 373) took 43ms (client timeout 50). W1103 05:09:54.847367 2622 inbound_call.cc:140] Trace: 1103 05:09:54.803729 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:54.803749 (+ 20us) service_pool.cc:152] Handling call 1103 05:09:54.803761 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:54.803763 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:54.803795 (+ 32us) log.cc:406] Serialized 496 byte log entry 1103 05:09:54.803803 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 376 1103 05:09:54.803815 (+ 12us) raft_consensus.cc:682] Updating last received op as term: 1 index: 377 1103 05:09:54.803817 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:54.847277 (+ 43460us) raft_consensus.cc:697] finished 1103 05:09:54.847278 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:54.847280 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:54.847285 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:54.803803 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:54.803810 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:54.803811 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:54.803813 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:54.803821 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:54.803831 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:54.803832 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:54.803834 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:54.803837 (+ 3us) write_transaction.cc:95] Timestamp: L: 750 W1103 05:09:54.918903 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.402-1.402] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:54.918933 2325 consensus_peers.cc:368] Couldn't send request to peer 6c5033bd4ec64c09b084bd11860c4d04 for tablet 607a992db198406ea346bc4267b364fa ops [1.402-1.402] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:54.919005 3589 log.cc:507] Time spent Append to log took a long time: real 0.050s user 0.000s sys 0.000s W1103 05:09:54.919059 2859 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 402) took 50ms (client timeout 50). W1103 05:09:54.919117 2859 inbound_call.cc:140] Trace: 1103 05:09:54.868702 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:54.868726 (+ 24us) service_pool.cc:152] Handling call 1103 05:09:54.868737 (+ 11us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:54.868738 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:54.868767 (+ 29us) log.cc:406] Serialized 496 byte log entry 1103 05:09:54.868775 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 401 1103 05:09:54.868790 (+ 15us) raft_consensus.cc:682] Updating last received op as term: 1 index: 402 1103 05:09:54.868791 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:54.919040 (+ 50249us) raft_consensus.cc:697] finished 1103 05:09:54.919041 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:54.919043 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:54.919048 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:54.868803 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:54.868807 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:54.868808 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:54.868809 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:54.868817 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:54.868825 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:54.868826 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:54.868827 (+ 1us) write_transaction.cc:93] Start() 1103 05:09:54.868830 (+ 3us) write_transaction.cc:95] Timestamp: L: 801 W1103 05:09:54.919850 3587 log.cc:507] Time spent Append to log took a long time: real 0.051s user 0.000s sys 0.000s W1103 05:09:54.919903 2624 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 395) took 51ms (client timeout 50). W1103 05:09:54.919945 2624 inbound_call.cc:140] Trace: 1103 05:09:54.868681 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:54.868709 (+ 28us) service_pool.cc:152] Handling call 1103 05:09:54.868723 (+ 14us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:54.868725 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:54.868749 (+ 24us) log.cc:406] Serialized 496 byte log entry 1103 05:09:54.868755 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 401 1103 05:09:54.868763 (+ 8us) raft_consensus.cc:682] Updating last received op as term: 1 index: 402 1103 05:09:54.868764 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:54.919884 (+ 51120us) raft_consensus.cc:697] finished 1103 05:09:54.919885 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:54.919887 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:54.919892 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:54.868764 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:54.868771 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:54.868772 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:54.868774 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:54.868786 (+ 12us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:54.868797 (+ 11us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:54.868799 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:54.868801 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:54.868804 (+ 3us) write_transaction.cc:95] Timestamp: L: 799 W1103 05:09:54.920084 3586 log.cc:507] Time spent Append to log took a long time: real 0.051s user 0.000s sys 0.000s W1103 05:09:55.503926 2859 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 452) took 40ms (client timeout 50). W1103 05:09:55.503993 2859 inbound_call.cc:140] Trace: 1103 05:09:55.463334 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:55.463361 (+ 27us) service_pool.cc:152] Handling call 1103 05:09:55.463373 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:55.463375 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:55.463405 (+ 30us) log.cc:406] Serialized 496 byte log entry 1103 05:09:55.463412 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 455 1103 05:09:55.463423 (+ 11us) raft_consensus.cc:682] Updating last received op as term: 1 index: 456 1103 05:09:55.463425 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:55.503898 (+ 40473us) raft_consensus.cc:697] finished 1103 05:09:55.503901 (+ 3us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:55.503903 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:55.503909 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:55.463436 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:55.463441 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:55.463442 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:55.463444 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:55.463452 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:55.463458 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:55.463459 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:55.463460 (+ 1us) write_transaction.cc:93] Start() 1103 05:09:55.463463 (+ 3us) write_transaction.cc:95] Timestamp: L: 907 W1103 05:09:55.555029 2325 consensus_peers.cc:368] Couldn't send request to peer 6c5033bd4ec64c09b084bd11860c4d04 for tablet 607a992db198406ea346bc4267b364fa ops [1.460-1.460] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:55.555202 2324 consensus_peers.cc:368] Couldn't send request to peer 07cb960abedc43a7a8bea75e1b465197 for tablet 607a992db198406ea346bc4267b364fa ops [1.460-1.460] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:09:55.573161 3587 log.cc:507] Time spent Append to log took a long time: real 0.068s user 0.000s sys 0.000s W1103 05:09:55.573210 2623 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:33633 (request call id 454) took 68ms (client timeout 50). W1103 05:09:55.573253 2623 inbound_call.cc:140] Trace: 1103 05:09:55.505055 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:55.505082 (+ 27us) service_pool.cc:152] Handling call 1103 05:09:55.505098 (+ 16us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:55.505100 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:55.505130 (+ 30us) log.cc:406] Serialized 496 byte log entry 1103 05:09:55.505138 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 459 1103 05:09:55.505156 (+ 18us) raft_consensus.cc:682] Updating last received op as term: 1 index: 460 1103 05:09:55.505158 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:55.573194 (+ 68036us) raft_consensus.cc:697] finished 1103 05:09:55.573195 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:55.573197 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:55.573201 (+ 4us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:55.505190 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:55.505195 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:55.505197 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:09:55.505198 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:55.505207 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:55.505227 (+ 20us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:55.505229 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:55.505231 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:55.505235 (+ 4us) write_transaction.cc:95] Timestamp: L: 915 W1103 05:09:55.581053 3589 log.cc:507] Time spent Append to log took a long time: real 0.076s user 0.000s sys 0.000s W1103 05:09:55.581131 2861 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:42405 (request call id 454) took 76ms (client timeout 50). W1103 05:09:55.581195 2861 inbound_call.cc:140] Trace: 1103 05:09:55.504826 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:09:55.504849 (+ 23us) service_pool.cc:152] Handling call 1103 05:09:55.504862 (+ 13us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:09:55.504865 (+ 3us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:09:55.504895 (+ 30us) log.cc:406] Serialized 496 byte log entry 1103 05:09:55.504904 (+ 9us) raft_consensus.cc:669] Marking committed up to term: 1 index: 459 1103 05:09:55.504908 (+ 4us) raft_consensus.cc:682] Updating last received op as term: 1 index: 460 1103 05:09:55.504910 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:09:55.581110 (+ 76200us) raft_consensus.cc:697] finished 1103 05:09:55.581112 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:09:55.581114 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:09:55.581120 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:09:55.504919 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:09:55.504925 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:09:55.504926 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:09:55.504927 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:09:55.504937 (+ 10us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:09:55.504947 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:09:55.504948 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:09:55.504950 (+ 2us) write_transaction.cc:93] Start() 1103 05:09:55.504953 (+ 3us) write_transaction.cc:95] Timestamp: L: 916 W1103 05:09:55.581368 3586 log.cc:507] Time spent Append to log took a long time: real 0.077s user 0.000s sys 0.000s W1103 05:09:56.037894 3695 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:36186, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.084676 3699 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:40929, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.149942 3695 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:36186, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.192601 3699 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:40929, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.262673 3695 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:36186, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.303980 3699 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:40929, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.381322 3695 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:36186, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.420619 3699 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:40929, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.508811 3695 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:36186, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.548176 3699 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:40929, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:09:56.597731 3701 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58567, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:09:56.649193 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master with pid 2173 I1103 05:09:56.651556 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 2267 I1103 05:09:56.655674 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 2410 I1103 05:09:56.660635 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 2636 [ OK ] DistConsensusTest.TestInsertAndMutateThroughConsensus (7925 ms) [ RUN ] DistConsensusTest.TestFailedTransaction I1103 05:09:56.668175 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master kudu-master --master_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --master_web_port=0 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:56.694862 6456 master_main.cc:27] Initializing master server... I1103 05:09:56.769520 6456 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/master/instance: uuid: "7d7a290d8164494f85aadb431de69500" format_stamp: "Formatted at 2014-11-03 13:09:56 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:56.769644 6456 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/master uuid: "7d7a290d8164494f85aadb431de69500" format_stamp: "Formatted at 2014-11-03 13:09:56 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:56.772512 6456 master_main.cc:30] Starting master server... I1103 05:09:56.774592 6456 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:56.774610 6456 webserver.cc:117] Document root disabled I1103 05:09:56.775097 6456 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:39158 I1103 05:09:56.775349 6456 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb I1103 05:09:56.779649 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:59174 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:56.801187 6546 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:56.869691 6546 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/instance: uuid: "5c0328f00b544edeb295b830825a1430" format_stamp: "Formatted at 2014-11-03 13:09:56 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:56.869822 6546 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 uuid: "5c0328f00b544edeb295b830825a1430" format_stamp: "Formatted at 2014-11-03 13:09:56 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:56.873136 6546 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:56.876355 6546 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:56.876385 6546 webserver.cc:117] Document root disabled I1103 05:09:56.876842 6546 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:59240 I1103 05:09:56.877029 6546 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb I1103 05:09:56.877149 6546 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:56.877379 6546 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:56.878592 6589 heartbeater.cc:227] Registering TS with master... I1103 05:09:56.878613 6589 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:56.886350 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:59174 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:56.907738 6599 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:56.976817 6599 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/instance: uuid: "c343df03e3924e10a9f80dc1f4628306" format_stamp: "Formatted at 2014-11-03 13:09:56 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:56.977382 6599 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 uuid: "c343df03e3924e10a9f80dc1f4628306" format_stamp: "Formatted at 2014-11-03 13:09:56 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:56.981034 6599 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:56.984154 6599 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:56.984187 6599 webserver.cc:117] Document root disabled I1103 05:09:56.984666 6599 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:33039 I1103 05:09:56.984885 6599 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb I1103 05:09:56.985210 6599 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:56.985419 6599 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:56.986639 6642 heartbeater.cc:227] Registering TS with master... I1103 05:09:56.986675 6642 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:56.993170 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:59174 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:09:57.019088 6645 tablet_server_main.cc:38] Initializing tablet server... I1103 05:09:57.029760 6545 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: Bootstrap starting. I1103 05:09:57.030225 6545 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one. I1103 05:09:57.030457 6545 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:09:57.030601 6651 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:09:57.030874 6545 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: No bootstrap required, opened a new log I1103 05:09:57.031960 6653 tablet_peer.cc:187] Configuration changed for tablet: 11111111111111111111111111111111 in peer: 7d7a290d8164494f85aadb431de69500 Changed from: seqno: 0 local: true peers { permanent_uuid: "7d7a290d8164494f85aadb431de69500" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "7d7a290d8164494f85aadb431de69500" role: LEADER } I1103 05:09:57.031991 6653 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "7d7a290d8164494f85aadb431de69500" role: LEADER } I1103 05:09:57.139675 6645 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/instance: uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" format_stamp: "Formatted at 2014-11-03 13:09:57 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:57.139803 6645 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" format_stamp: "Formatted at 2014-11-03 13:09:57 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:09:57.143044 6645 tablet_server_main.cc:41] Starting tablet server... I1103 05:09:57.146286 6645 webserver.cc:108] Starting webserver on localhost:0 I1103 05:09:57.146317 6645 webserver.cc:117] Document root disabled I1103 05:09:57.146844 6645 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:37778 I1103 05:09:57.147059 6645 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestFailedTransaction.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb I1103 05:09:57.147194 6645 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:09:57.147341 6645 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:09:57.148324 6703 heartbeater.cc:227] Registering TS with master... I1103 05:09:57.148362 6703 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:57.388581 6545 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: Bootstrap starting. I1103 05:09:57.388996 6545 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I1103 05:09:57.389503 6545 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I1103 05:09:57.389931 6545 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. W1103 05:09:57.389983 6522 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:09:57.390005 6524 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:09:57.390053 6526 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. I1103 05:09:57.390461 6726 tablet_peer.cc:187] Configuration changed for tablet: 00000000000000000000000000000000 in peer: 7d7a290d8164494f85aadb431de69500 Changed from: seqno: 0 local: true peers { permanent_uuid: "7d7a290d8164494f85aadb431de69500" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "7d7a290d8164494f85aadb431de69500" role: LEADER } I1103 05:09:57.390486 6726 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "7d7a290d8164494f85aadb431de69500" role: LEADER } I1103 05:09:57.390496 6545 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:09:57.390600 6456 master_main.cc:33] Master server successfully started. I1103 05:09:58.390697 6528 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" instance_seqno: 1415020196983952 } as {real_user=jenkins, eff_user=} at 127.0.0.1:53385; Asking this server to re-register. I1103 05:09:58.390702 6529 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "5c0328f00b544edeb295b830825a1430" instance_seqno: 1415020196876219 } as {real_user=jenkins, eff_user=} at 127.0.0.1:53384; Asking this server to re-register. I1103 05:09:58.390810 6530 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" instance_seqno: 1415020197146147 } as {real_user=jenkins, eff_user=} at 127.0.0.1:53386; Asking this server to re-register. I1103 05:09:58.391033 6589 heartbeater.cc:227] Registering TS with master... I1103 05:09:58.391059 6589 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:58.391100 6703 heartbeater.cc:227] Registering TS with master... I1103 05:09:58.391122 6703 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:58.391408 6525 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" instance_seqno: 1415020197146147 } with Master I1103 05:09:58.391571 6523 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "5c0328f00b544edeb295b830825a1430" instance_seqno: 1415020196876219 } with Master I1103 05:09:58.391768 6642 heartbeater.cc:227] Registering TS with master... I1103 05:09:58.391793 6642 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:09:58.392176 6522 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" instance_seqno: 1415020196983952 } with Master I1103 05:09:58.392719 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:09:58.393064 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:09:58.396824 6528 catalog_manager.cc:437] CreateTable from {real_user=jenkins, eff_user=} at 127.0.0.1:53390: name: "TestTable" schema { columns { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I1103 05:09:58.399626 6620 tablet_service.cc:264] Processing CreateTablet for tablet 3c8da462164a4ad0ba582ba2dcb84739 (table=TestTable [id=37d3ebc486854a1d93754c0f79d0692b]), range=["", ""] I1103 05:09:58.400163 6567 tablet_service.cc:264] Processing CreateTablet for tablet 3c8da462164a4ad0ba582ba2dcb84739 (table=TestTable [id=37d3ebc486854a1d93754c0f79d0692b]), range=["", ""] I1103 05:09:58.400668 6681 tablet_service.cc:264] Processing CreateTablet for tablet 3c8da462164a4ad0ba582ba2dcb84739 (table=TestTable [id=37d3ebc486854a1d93754c0f79d0692b]), range=["", ""] I1103 05:09:59.881127 6681 ts_tablet_manager.cc:464] Registered tablet 3c8da462164a4ad0ba582ba2dcb84739 I1103 05:09:59.881136 6567 ts_tablet_manager.cc:464] Registered tablet 3c8da462164a4ad0ba582ba2dcb84739 I1103 05:09:59.881151 6620 ts_tablet_manager.cc:464] Registered tablet 3c8da462164a4ad0ba582ba2dcb84739 I1103 05:09:59.881217 7292 ts_tablet_manager.cc:325] Bootstrapping tablet: 3c8da462164a4ad0ba582ba2dcb84739 I1103 05:09:59.881265 7291 ts_tablet_manager.cc:325] Bootstrapping tablet: 3c8da462164a4ad0ba582ba2dcb84739 I1103 05:09:59.881274 7293 ts_tablet_manager.cc:325] Bootstrapping tablet: 3c8da462164a4ad0ba582ba2dcb84739 I1103 05:09:59.881418 7291 tablet_bootstrap.cc:234] Tablet 3c8da462164a4ad0ba582ba2dcb84739: Bootstrap starting. I1103 05:09:59.881476 7293 tablet_bootstrap.cc:234] Tablet 3c8da462164a4ad0ba582ba2dcb84739: Bootstrap starting. I1103 05:09:59.881530 7292 tablet_bootstrap.cc:234] Tablet 3c8da462164a4ad0ba582ba2dcb84739: Bootstrap starting. I1103 05:09:59.882302 7291 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 3c8da462164a4ad0ba582ba2dcb84739 creating new one. I1103 05:09:59.882483 7292 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 3c8da462164a4ad0ba582ba2dcb84739 creating new one. I1103 05:09:59.882503 7293 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 3c8da462164a4ad0ba582ba2dcb84739 creating new one. I1103 05:09:59.882647 7291 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:09:59.882803 7292 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:09:59.882805 7293 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:09:59.882990 7299 env_posix.cc:511] The filesystem does not support fallocate(). W1103 05:09:59.883004 7297 env_posix.cc:511] The filesystem does not support fallocate(). W1103 05:09:59.883106 7298 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:09:59.883221 7291 tablet_bootstrap.cc:234] Tablet 3c8da462164a4ad0ba582ba2dcb84739: No bootstrap required, opened a new log I1103 05:09:59.883240 7292 tablet_bootstrap.cc:234] Tablet 3c8da462164a4ad0ba582ba2dcb84739: No bootstrap required, opened a new log I1103 05:09:59.883270 7291 ts_tablet_manager.cc:330] Time spent tablet 3c8da462164a4ad0ba582ba2dcb84739 bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:09:59.883307 7292 ts_tablet_manager.cc:330] Time spent tablet 3c8da462164a4ad0ba582ba2dcb84739 bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:09:59.883353 7293 tablet_bootstrap.cc:234] Tablet 3c8da462164a4ad0ba582ba2dcb84739: No bootstrap required, opened a new log I1103 05:09:59.883414 7293 ts_tablet_manager.cc:330] Time spent tablet 3c8da462164a4ad0ba582ba2dcb84739 bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:09:59.883616 7291 raft_consensus.cc:102] T 3c8da462164a4ad0ba582ba2dcb84739 P 3c05ee8fc5eb4dec9c4f0b19927f6ece [CANDIDATE]: Replica starting. Triggering 0 pending transactions. I1103 05:09:59.883744 7292 raft_consensus.cc:102] T 3c8da462164a4ad0ba582ba2dcb84739 P 5c0328f00b544edeb295b830825a1430 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:09:59.883834 7293 raft_consensus.cc:102] T 3c8da462164a4ad0ba582ba2dcb84739 P c343df03e3924e10a9f80dc1f4628306 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:09:59.883903 7292 raft_consensus.cc:302] T 3c8da462164a4ad0ba582ba2dcb84739 P 5c0328f00b544edeb295b830825a1430 [FOLLOWER]: Becoming Follower/Learner I1103 05:09:59.883898 7291 raft_consensus.cc:235] T 3c8da462164a4ad0ba582ba2dcb84739 P 3c05ee8fc5eb4dec9c4f0b19927f6ece [LEADER]: Becoming Leader I1103 05:09:59.884114 7293 raft_consensus.cc:302] T 3c8da462164a4ad0ba582ba2dcb84739 P c343df03e3924e10a9f80dc1f4628306 [FOLLOWER]: Becoming Follower/Learner I1103 05:09:59.884192 7293 ts_tablet_manager.cc:350] Time spent starting tablet 3c8da462164a4ad0ba582ba2dcb84739: real 0.001s user 0.000s sys 0.000s I1103 05:09:59.884399 7292 ts_tablet_manager.cc:350] Time spent starting tablet 3c8da462164a4ad0ba582ba2dcb84739: real 0.001s user 0.000s sys 0.001s I1103 05:09:59.885555 7291 ts_tablet_manager.cc:350] Time spent starting tablet 3c8da462164a4ad0ba582ba2dcb84739: real 0.002s user 0.002s sys 0.000s W1103 05:10:00.384763 7305 raft_consensus_state.cc:463] T 3c8da462164a4ad0ba582ba2dcb84739 P 3c05ee8fc5eb4dec9c4f0b19927f6ece [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 W1103 05:10:00.385081 7300 raft_consensus_state.cc:463] T 3c8da462164a4ad0ba582ba2dcb84739 P 3c05ee8fc5eb4dec9c4f0b19927f6ece [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:10:00.386989 6577 raft_consensus.cc:608] T 3c8da462164a4ad0ba582ba2dcb84739 P 5c0328f00b544edeb295b830825a1430 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:00.387097 6630 raft_consensus.cc:608] T 3c8da462164a4ad0ba582ba2dcb84739 P c343df03e3924e10a9f80dc1f4628306 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:00.387131 6667 consensus_queue.cc:451] Connected to peer: 5c0328f00b544edeb295b830825a1430 whose last received watermark is: term: 0 index: 0 I1103 05:10:00.387280 6666 consensus_queue.cc:451] Connected to peer: c343df03e3924e10a9f80dc1f4628306 whose last received watermark is: term: 0 index: 0 I1103 05:10:00.388268 6667 raft_consensus_state.cc:456] T 3c8da462164a4ad0ba582ba2dcb84739 P 3c05ee8fc5eb4dec9c4f0b19927f6ece [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I1103 05:10:00.388382 7300 tablet_peer.cc:187] Configuration changed for tablet: 3c8da462164a4ad0ba582ba2dcb84739 in peer: 3c05ee8fc5eb4dec9c4f0b19927f6ece Changed from: seqno: -1 local: false peers { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43602 } } peers { permanent_uuid: "5c0328f00b544edeb295b830825a1430" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40835 } } peers { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 58630 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43602 } } peers { permanent_uuid: "5c0328f00b544edeb295b830825a1430" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40835 } } peers { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" role: LEADER last_known_addr { host: "127.0.0.1" port: 58630 } } I1103 05:10:00.392336 6529 catalog_manager.cc:1127] Tablet: 3c8da462164a4ad0ba582ba2dcb84739 reported quorum change. New quorum: seqno: 0 local: false peers { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43602 } } peers { permanent_uuid: "5c0328f00b544edeb295b830825a1430" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40835 } } peers { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" role: LEADER last_known_addr { host: "127.0.0.1" port: 58630 } } I1103 05:10:00.469414 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master W1103 05:10:00.469553 6447 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:59174, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:00.474148 7302 tablet_peer.cc:187] Configuration changed for tablet: 3c8da462164a4ad0ba582ba2dcb84739 in peer: c343df03e3924e10a9f80dc1f4628306 Changed from: seqno: -1 local: false peers { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43602 } } peers { permanent_uuid: "5c0328f00b544edeb295b830825a1430" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40835 } } peers { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 58630 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43602 } } peers { permanent_uuid: "5c0328f00b544edeb295b830825a1430" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40835 } } peers { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" role: LEADER last_known_addr { host: "127.0.0.1" port: 58630 } } I1103 05:10:00.474298 7301 tablet_peer.cc:187] Configuration changed for tablet: 3c8da462164a4ad0ba582ba2dcb84739 in peer: 5c0328f00b544edeb295b830825a1430 Changed from: seqno: -1 local: false peers { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43602 } } peers { permanent_uuid: "5c0328f00b544edeb295b830825a1430" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40835 } } peers { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 58630 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "c343df03e3924e10a9f80dc1f4628306" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43602 } } peers { permanent_uuid: "5c0328f00b544edeb295b830825a1430" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 40835 } } peers { permanent_uuid: "3c05ee8fc5eb4dec9c4f0b19927f6ece" role: LEADER last_known_addr { host: "127.0.0.1" port: 58630 } } W1103 05:10:00.475136 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.476390 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.488349 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.489696 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.510303 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.511252 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.541746 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.542678 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.583353 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.584421 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.635057 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.635700 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.696259 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.697108 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.767869 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.768749 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.849427 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.850208 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.940827 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:00.941613 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:01.042245 7366 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58630, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:01.042981 7367 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43602, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:01.043771 7368 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:40835, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:01.045897 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master with pid 6456 I1103 05:10:01.048511 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 6546 I1103 05:10:01.051512 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 6599 I1103 05:10:01.055261 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 6645 [ OK ] DistConsensusTest.TestFailedTransaction (4395 ms) [ RUN ] DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus I1103 05:10:01.063379 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master kudu-master --master_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --master_web_port=0 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:01.089853 7493 master_main.cc:27] Initializing master server... I1103 05:10:01.588037 7493 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master/instance: uuid: "d37af8630a474d2a815ad2553ee0d3e1" format_stamp: "Formatted at 2014-11-03 13:10:01 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:01.588183 7493 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master uuid: "d37af8630a474d2a815ad2553ee0d3e1" format_stamp: "Formatted at 2014-11-03 13:10:01 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:01.592342 7493 master_main.cc:30] Starting master server... I1103 05:10:01.596696 7493 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:01.596719 7493 webserver.cc:117] Document root disabled I1103 05:10:01.597245 7493 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:50152 I1103 05:10:01.597435 7493 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb I1103 05:10:01.606892 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:53185 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:01.628165 7545 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:01.785923 7545 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/instance: uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" format_stamp: "Formatted at 2014-11-03 13:10:01 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:01.786074 7545 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" format_stamp: "Formatted at 2014-11-03 13:10:01 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:01.789404 7545 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:01.792654 7545 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:01.792687 7545 webserver.cc:117] Document root disabled I1103 05:10:01.793208 7545 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:44147 I1103 05:10:01.793438 7545 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb I1103 05:10:01.793553 7545 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:01.793653 7545 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:01.794962 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:53185 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:01.796629 7591 heartbeater.cc:227] Registering TS with master... I1103 05:10:01.796655 7591 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:01.818174 7595 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:01.933228 7544 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: Bootstrap starting. I1103 05:10:01.933603 7544 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one. I1103 05:10:01.933841 7544 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:01.934003 7704 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:01.934227 7544 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: No bootstrap required, opened a new log I1103 05:10:01.935077 7705 tablet_peer.cc:187] Configuration changed for tablet: 11111111111111111111111111111111 in peer: d37af8630a474d2a815ad2553ee0d3e1 Changed from: seqno: 0 local: true peers { permanent_uuid: "d37af8630a474d2a815ad2553ee0d3e1" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "d37af8630a474d2a815ad2553ee0d3e1" role: LEADER } I1103 05:10:01.935103 7705 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "d37af8630a474d2a815ad2553ee0d3e1" role: LEADER } I1103 05:10:01.964177 7595 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/instance: uuid: "20535f6222dc4aab9c9d9b5858d00404" format_stamp: "Formatted at 2014-11-03 13:10:01 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:01.964308 7595 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 uuid: "20535f6222dc4aab9c9d9b5858d00404" format_stamp: "Formatted at 2014-11-03 13:10:01 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:01.967298 7595 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:01.970270 7595 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:01.970319 7595 webserver.cc:117] Document root disabled I1103 05:10:01.970798 7595 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:34192 I1103 05:10:01.971006 7595 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb I1103 05:10:01.971140 7595 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:01.971243 7595 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:01.972333 7747 heartbeater.cc:227] Registering TS with master... I1103 05:10:01.972362 7747 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:01.972884 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:53185 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:02.000862 7750 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:02.126515 7750 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/instance: uuid: "e95cd7e351fc4e28bac7f7977928f571" format_stamp: "Formatted at 2014-11-03 13:10:02 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:02.126689 7750 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 uuid: "e95cd7e351fc4e28bac7f7977928f571" format_stamp: "Formatted at 2014-11-03 13:10:02 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:02.130420 7750 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:02.134234 7750 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:02.134273 7750 webserver.cc:117] Document root disabled I1103 05:10:02.134976 7750 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:53176 I1103 05:10:02.136749 7750 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb I1103 05:10:02.137320 7750 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:02.137660 7750 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:02.138715 7790 heartbeater.cc:227] Registering TS with master... I1103 05:10:02.138751 7790 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:02.152201 7544 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: Bootstrap starting. I1103 05:10:02.152658 7544 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I1103 05:10:02.153530 7544 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I1103 05:10:02.154258 7544 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. W1103 05:10:02.154294 7527 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:10:02.154391 7523 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:10:02.154414 7525 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. I1103 05:10:02.155092 7795 tablet_peer.cc:187] Configuration changed for tablet: 00000000000000000000000000000000 in peer: d37af8630a474d2a815ad2553ee0d3e1 Changed from: seqno: 0 local: true peers { permanent_uuid: "d37af8630a474d2a815ad2553ee0d3e1" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "d37af8630a474d2a815ad2553ee0d3e1" role: LEADER } I1103 05:10:02.155151 7544 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:10:02.155267 7493 master_main.cc:33] Master server successfully started. I1103 05:10:02.155313 7795 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "d37af8630a474d2a815ad2553ee0d3e1" role: LEADER } I1103 05:10:03.155167 7525 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" instance_seqno: 1415020201970135 } as {real_user=jenkins, eff_user=} at 127.0.0.1:51177; Asking this server to re-register. I1103 05:10:03.155169 7523 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" instance_seqno: 1415020202134082 } as {real_user=jenkins, eff_user=} at 127.0.0.1:51178; Asking this server to re-register. I1103 05:10:03.155167 7528 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" instance_seqno: 1415020201792511 } as {real_user=jenkins, eff_user=} at 127.0.0.1:51164; Asking this server to re-register. I1103 05:10:03.155489 7747 heartbeater.cc:227] Registering TS with master... I1103 05:10:03.155515 7747 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:03.155513 7790 heartbeater.cc:227] Registering TS with master... I1103 05:10:03.155539 7790 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:03.155551 7591 heartbeater.cc:227] Registering TS with master... I1103 05:10:03.155570 7591 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:03.155721 7530 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" instance_seqno: 1415020202134082 } with Master I1103 05:10:03.155753 7531 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" instance_seqno: 1415020201970135 } with Master I1103 05:10:03.155793 7522 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" instance_seqno: 1415020201792511 } with Master I1103 05:10:03.157137 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:10:03.157487 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:10:03.159842 7525 catalog_manager.cc:437] CreateTable from {real_user=jenkins, eff_user=} at 127.0.0.1:51191: name: "TestTable" schema { columns { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I1103 05:10:03.175340 7569 tablet_service.cc:264] Processing CreateTablet for tablet 69ce28a55ab34dafb931a4677a97f8ae (table=TestTable [id=ced1574a54eb4ee59d4252b7bdbf52e3]), range=["", ""] I1103 05:10:03.175585 7768 tablet_service.cc:264] Processing CreateTablet for tablet 69ce28a55ab34dafb931a4677a97f8ae (table=TestTable [id=ced1574a54eb4ee59d4252b7bdbf52e3]), range=["", ""] I1103 05:10:03.176126 7725 tablet_service.cc:264] Processing CreateTablet for tablet 69ce28a55ab34dafb931a4677a97f8ae (table=TestTable [id=ced1574a54eb4ee59d4252b7bdbf52e3]), range=["", ""] I1103 05:10:03.479761 7768 ts_tablet_manager.cc:464] Registered tablet 69ce28a55ab34dafb931a4677a97f8ae I1103 05:10:03.479921 7569 ts_tablet_manager.cc:464] Registered tablet 69ce28a55ab34dafb931a4677a97f8ae I1103 05:10:03.479969 8080 ts_tablet_manager.cc:325] Bootstrapping tablet: 69ce28a55ab34dafb931a4677a97f8ae I1103 05:10:03.480106 8080 tablet_bootstrap.cc:234] Tablet 69ce28a55ab34dafb931a4677a97f8ae: Bootstrap starting. I1103 05:10:03.480193 8081 ts_tablet_manager.cc:325] Bootstrapping tablet: 69ce28a55ab34dafb931a4677a97f8ae I1103 05:10:03.480376 8081 tablet_bootstrap.cc:234] Tablet 69ce28a55ab34dafb931a4677a97f8ae: Bootstrap starting. I1103 05:10:03.480496 8080 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 69ce28a55ab34dafb931a4677a97f8ae creating new one. I1103 05:10:03.480758 8080 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:10:03.480758 8081 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 69ce28a55ab34dafb931a4677a97f8ae creating new one. W1103 05:10:03.480947 8082 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:03.481003 8081 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:03.481171 8083 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:03.481328 8080 tablet_bootstrap.cc:234] Tablet 69ce28a55ab34dafb931a4677a97f8ae: No bootstrap required, opened a new log I1103 05:10:03.481379 8080 ts_tablet_manager.cc:330] Time spent tablet 69ce28a55ab34dafb931a4677a97f8ae bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:10:03.481539 8081 tablet_bootstrap.cc:234] Tablet 69ce28a55ab34dafb931a4677a97f8ae: No bootstrap required, opened a new log I1103 05:10:03.481570 8081 ts_tablet_manager.cc:330] Time spent tablet 69ce28a55ab34dafb931a4677a97f8ae bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:10:03.481719 8080 raft_consensus.cc:102] T 69ce28a55ab34dafb931a4677a97f8ae P e95cd7e351fc4e28bac7f7977928f571 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:03.481828 8080 raft_consensus.cc:302] T 69ce28a55ab34dafb931a4677a97f8ae P e95cd7e351fc4e28bac7f7977928f571 [FOLLOWER]: Becoming Follower/Learner I1103 05:10:03.481865 8080 ts_tablet_manager.cc:350] Time spent starting tablet 69ce28a55ab34dafb931a4677a97f8ae: real 0.000s user 0.000s sys 0.000s I1103 05:10:03.481864 8081 raft_consensus.cc:102] T 69ce28a55ab34dafb931a4677a97f8ae P 1135cdf4560a4f92ba4d3cb0c17f4eff [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:03.481963 8081 raft_consensus.cc:302] T 69ce28a55ab34dafb931a4677a97f8ae P 1135cdf4560a4f92ba4d3cb0c17f4eff [FOLLOWER]: Becoming Follower/Learner I1103 05:10:03.482009 8081 ts_tablet_manager.cc:350] Time spent starting tablet 69ce28a55ab34dafb931a4677a97f8ae: real 0.000s user 0.000s sys 0.000s I1103 05:10:03.649857 7725 ts_tablet_manager.cc:464] Registered tablet 69ce28a55ab34dafb931a4677a97f8ae I1103 05:10:03.650161 8091 ts_tablet_manager.cc:325] Bootstrapping tablet: 69ce28a55ab34dafb931a4677a97f8ae I1103 05:10:03.650305 8091 tablet_bootstrap.cc:234] Tablet 69ce28a55ab34dafb931a4677a97f8ae: Bootstrap starting. I1103 05:10:03.650820 8091 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 69ce28a55ab34dafb931a4677a97f8ae creating new one. I1103 05:10:03.651140 8091 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:03.651466 8092 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:03.651938 8091 tablet_bootstrap.cc:234] Tablet 69ce28a55ab34dafb931a4677a97f8ae: No bootstrap required, opened a new log I1103 05:10:03.652003 8091 ts_tablet_manager.cc:330] Time spent tablet 69ce28a55ab34dafb931a4677a97f8ae bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:10:03.652449 8091 raft_consensus.cc:102] T 69ce28a55ab34dafb931a4677a97f8ae P 20535f6222dc4aab9c9d9b5858d00404 [CANDIDATE]: Replica starting. Triggering 0 pending transactions. I1103 05:10:03.653339 8091 raft_consensus.cc:235] T 69ce28a55ab34dafb931a4677a97f8ae P 20535f6222dc4aab9c9d9b5858d00404 [LEADER]: Becoming Leader I1103 05:10:03.656474 8091 ts_tablet_manager.cc:350] Time spent starting tablet 69ce28a55ab34dafb931a4677a97f8ae: real 0.004s user 0.002s sys 0.001s W1103 05:10:04.154175 8096 raft_consensus_state.cc:463] T 69ce28a55ab34dafb931a4677a97f8ae P 20535f6222dc4aab9c9d9b5858d00404 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 W1103 05:10:04.154623 8093 raft_consensus_state.cc:463] T 69ce28a55ab34dafb931a4677a97f8ae P 20535f6222dc4aab9c9d9b5858d00404 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:10:04.155943 7778 raft_consensus.cc:608] T 69ce28a55ab34dafb931a4677a97f8ae P e95cd7e351fc4e28bac7f7977928f571 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:04.156169 7712 consensus_queue.cc:451] Connected to peer: e95cd7e351fc4e28bac7f7977928f571 whose last received watermark is: term: 0 index: 0 I1103 05:10:04.156805 7579 raft_consensus.cc:608] T 69ce28a55ab34dafb931a4677a97f8ae P 1135cdf4560a4f92ba4d3cb0c17f4eff [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:04.156975 7710 consensus_queue.cc:451] Connected to peer: 1135cdf4560a4f92ba4d3cb0c17f4eff whose last received watermark is: term: 0 index: 0 I1103 05:10:04.157351 7712 raft_consensus_state.cc:456] T 69ce28a55ab34dafb931a4677a97f8ae P 20535f6222dc4aab9c9d9b5858d00404 [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I1103 05:10:04.157471 8093 tablet_peer.cc:187] Configuration changed for tablet: 69ce28a55ab34dafb931a4677a97f8ae in peer: 20535f6222dc4aab9c9d9b5858d00404 Changed from: seqno: -1 local: false peers { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58392 } } peers { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 42739 } } peers { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 54764 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58392 } } peers { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 42739 } } peers { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" role: LEADER last_known_addr { host: "127.0.0.1" port: 54764 } } I1103 05:10:04.236073 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master W1103 05:10:04.236192 7492 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:53185, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:04.237135 31061 raft_consensus-itest.cc:232] Couldn't find the leader and/or replicas. Locations: tablet_id: "69ce28a55ab34dafb931a4677a97f8ae" start_key: "" end_key: "" replicas { ts_info { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" rpc_addresses { host: "127.0.0.1" port: 58392 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" rpc_addresses { host: "127.0.0.1" port: 42739 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" rpc_addresses { host: "127.0.0.1" port: 54764 } } role: CANDIDATE } stale: false I1103 05:10:04.657002 8084 tablet_peer.cc:187] Configuration changed for tablet: 69ce28a55ab34dafb931a4677a97f8ae in peer: e95cd7e351fc4e28bac7f7977928f571 Changed from: seqno: -1 local: false peers { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58392 } } peers { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 42739 } } peers { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 54764 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58392 } } peers { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 42739 } } peers { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" role: LEADER last_known_addr { host: "127.0.0.1" port: 54764 } } I1103 05:10:04.659075 8085 tablet_peer.cc:187] Configuration changed for tablet: 69ce28a55ab34dafb931a4677a97f8ae in peer: 1135cdf4560a4f92ba4d3cb0c17f4eff Changed from: seqno: -1 local: false peers { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58392 } } peers { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 42739 } } peers { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 54764 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58392 } } peers { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 42739 } } peers { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" role: LEADER last_known_addr { host: "127.0.0.1" port: 54764 } } I1103 05:10:05.157574 7530 catalog_manager.cc:1127] Tablet: 69ce28a55ab34dafb931a4677a97f8ae reported quorum change. New quorum: seqno: 0 local: false peers { permanent_uuid: "e95cd7e351fc4e28bac7f7977928f571" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58392 } } peers { permanent_uuid: "1135cdf4560a4f92ba4d3cb0c17f4eff" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 42739 } } peers { permanent_uuid: "20535f6222dc4aab9c9d9b5858d00404" role: LEADER last_known_addr { host: "127.0.0.1" port: 54764 } } W1103 05:10:05.237442 7492 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:53185, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:05.248286 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 33278.4 usec... I1103 05:10:05.261454 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 18655.4 usec... I1103 05:10:05.314262 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15575.4 usec... I1103 05:10:05.314456 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 66623.9 usec... I1103 05:10:05.316329 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 23911.4 usec... I1103 05:10:05.330438 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 50611 usec... I1103 05:10:05.340708 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 2738.94 usec... I1103 05:10:05.343847 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 35549.3 usec... W1103 05:10:05.381477 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.119-1.120] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:05.381510 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 16322.3 usec... I1103 05:10:05.382879 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 19469.2 usec... I1103 05:10:05.398123 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 22391.8 usec... I1103 05:10:05.404355 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16922.4 usec... W1103 05:10:05.420812 8093 log.cc:507] Time spent Append to log took a long time: real 0.106s user 0.000s sys 0.000s I1103 05:10:05.420918 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 9200.85 usec... I1103 05:10:05.421493 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 3534.46 usec... I1103 05:10:05.430657 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 40471.3 usec... I1103 05:10:05.447414 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 1902.65 usec... I1103 05:10:05.449527 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 69646.9 usec... W1103 05:10:05.471575 8093 log.cc:507] Time spent Append to log took a long time: real 0.051s user 0.000s sys 0.000s I1103 05:10:05.477931 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 35558 usec... I1103 05:10:05.487354 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 11575.5 usec... I1103 05:10:05.499236 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 7713.97 usec... I1103 05:10:05.507182 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 19207.3 usec... I1103 05:10:05.519407 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 346.918 usec... I1103 05:10:05.525550 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 4789.25 usec... W1103 05:10:05.526603 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.119-1.122] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:05.561713 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 17658.2 usec... I1103 05:10:05.568759 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 6970.06 usec... I1103 05:10:05.579823 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 37826.3 usec... I1103 05:10:05.605445 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 9002.11 usec... I1103 05:10:05.614735 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 14692 usec... I1103 05:10:05.617933 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 39149 usec... W1103 05:10:05.657413 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.272-1.273] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:05.657461 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.268-1.269] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:05.657860 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 36328.5 usec... I1103 05:10:05.658298 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 20051.3 usec... I1103 05:10:05.678581 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 40525.1 usec... I1103 05:10:05.694998 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 38323.7 usec... I1103 05:10:05.733590 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 7324.74 usec... I1103 05:10:05.741147 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 28441.9 usec... I1103 05:10:05.764405 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 17865.5 usec... W1103 05:10:05.770115 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.275-1.275] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:05.782484 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 5635.31 usec... I1103 05:10:05.788579 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 17607.2 usec... I1103 05:10:05.806419 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16945.1 usec... W1103 05:10:05.821347 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.275-1.281] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:05.825373 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 73186 usec... I1103 05:10:05.859587 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 19812.9 usec... W1103 05:10:05.879902 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.275-1.443] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 3 times. I1103 05:10:05.899359 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 46453.2 usec... I1103 05:10:05.902425 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 3306.3 usec... I1103 05:10:05.906069 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 13606.7 usec... I1103 05:10:05.919931 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 33633.5 usec... I1103 05:10:05.938601 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 40527.1 usec... I1103 05:10:05.946182 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 12345.7 usec... W1103 05:10:05.953812 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.275-1.538] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 4 times. I1103 05:10:05.954082 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 64644.6 usec... W1103 05:10:05.958758 8085 log.cc:507] Time spent Append to log took a long time: real 0.074s user 0.000s sys 0.000s W1103 05:10:05.958788 7586 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 102) took 59ms (client timeout 50). W1103 05:10:05.958843 7586 inbound_call.cc:140] Trace: 1103 05:10:05.899344 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:05.958763 (+ 59419us) service_pool.cc:137] Skipping call since client already timed out 1103 05:10:05.958771 (+ 8us) inbound_call.cc:83] Queueing failure response W1103 05:10:05.958866 7582 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 100) took 176ms (client timeout 50). W1103 05:10:05.958953 7582 inbound_call.cc:140] Trace: 1103 05:10:05.782493 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:05.788333 (+ 5840us) service_pool.cc:152] Handling call 1103 05:10:05.825294 (+ 36961us) raft_consensus.cc:579] Updating replica for 7 ops 1103 05:10:05.825298 (+ 4us) raft_consensus.cc:614] Triggering prepare for 6 ops 1103 05:10:05.825414 (+ 116us) log.cc:406] Serialized 3006 byte log entry 1103 05:10:05.825423 (+ 9us) raft_consensus.cc:669] Marking committed up to term: 1 index: 274 1103 05:10:05.825427 (+ 4us) raft_consensus.cc:682] Updating last received op as term: 1 index: 281 1103 05:10:05.825428 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:05.958823 (+133395us) raft_consensus.cc:697] finished 1103 05:10:05.958825 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:05.958828 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:10:05.958855 (+ 27us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:05.825441 (+ 0us) write_transaction.cc:62] PREPARE: Starting Related trace: Related trace: Related trace: Related trace: Related trace: I1103 05:10:05.964189 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 25900.3 usec... I1103 05:10:05.979643 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 8983.3 usec... I1103 05:10:05.989261 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 28064.5 usec... W1103 05:10:05.990697 7588 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 101) took 165ms (client timeout 50). W1103 05:10:05.998251 7588 inbound_call.cc:140] Trace: 1103 05:10:05.824812 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:05.824826 (+ 14us) service_pool.cc:152] Handling call 1103 05:10:05.959236 (+134410us) raft_consensus.cc:579] Updating replica for 169 ops 1103 05:10:05.959242 (+ 6us) raft_consensus.cc:614] Triggering prepare for 162 ops 1103 05:10:05.961948 (+ 2706us) log.cc:406] Serialized 81362 byte log entry 1103 05:10:05.961960 (+ 12us) raft_consensus.cc:669] Marking committed up to term: 1 index: 441 1103 05:10:05.962142 (+ 182us) raft_consensus.cc:682] Updating last received op as term: 1 index: 443 1103 05:10:05.962147 (+ 5us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:05.990676 (+ 28529us) raft_consensus.cc:697] finished 1103 05:10:05.990677 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:05.990679 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:05.990685 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:05.960433 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960436 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960437 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960438 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960445 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960454 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960456 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960457 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960460 (+ 3us) write_transaction.cc:95] Timestamp: L: 510 1103 05:10:05.962200 (+ 1740us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962203 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962205 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962213 (+ 8us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962215 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962218 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962220 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962223 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962225 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962227 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962229 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962230 (+ 1us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962233 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962237 (+ 4us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962809 (+ 572us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962811 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962813 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960462 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960465 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960467 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960468 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960476 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960483 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960485 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960486 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960489 (+ 3us) write_transaction.cc:95] Timestamp: L: 511 1103 05:10:05.962211 (+ 1722us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962215 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962217 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962219 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962222 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962225 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962227 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962229 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962231 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962233 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962235 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962236 (+ 1us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962241 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962250 (+ 9us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962824 (+ 574us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962826 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962828 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960491 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960494 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960495 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960497 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960503 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960523 (+ 20us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960524 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960526 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:05.960528 (+ 2us) write_transaction.cc:95] Timestamp: L: 512 1103 05:10:05.962236 (+ 1708us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962240 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962243 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962246 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962250 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962255 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962257 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962261 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962264 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962267 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962277 (+ 10us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962280 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962285 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962290 (+ 5us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962846 (+ 556us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962849 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962852 (+ 3us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960530 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960533 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960534 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960535 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960542 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960546 (+ 4us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960547 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960548 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960549 (+ 1us) write_transaction.cc:95] Timestamp: L: 513 1103 05:10:05.962240 (+ 1691us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962242 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962245 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962247 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962250 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962252 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962254 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962256 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962258 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962260 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962263 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962264 (+ 1us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962267 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962269 (+ 2us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962836 (+ 567us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962837 (+ 1us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962839 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960551 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960553 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960554 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960555 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960559 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960564 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960565 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960566 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960568 (+ 2us) write_transaction.cc:95] Timestamp: L: 514 1103 05:10:05.962252 (+ 1684us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962255 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962258 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962261 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962263 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962266 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962274 (+ 8us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962276 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962278 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962280 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962283 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962285 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962287 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962290 (+ 3us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962857 (+ 567us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962859 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962860 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960569 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960571 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960572 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960573 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960577 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960582 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960583 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960584 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960586 (+ 2us) write_transaction.cc:95] Timestamp: L: 515 1103 05:10:05.962272 (+ 1686us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962274 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962277 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962279 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962281 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962284 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962286 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962287 (+ 1us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962289 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962291 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962293 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962294 (+ 1us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962297 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962300 (+ 3us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962866 (+ 566us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962868 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962869 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960587 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960589 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960590 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960591 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960595 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960601 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960602 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960603 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960605 (+ 2us) write_transaction.cc:95] Timestamp: L: 516 1103 05:10:05.962296 (+ 1691us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962301 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962304 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962307 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962311 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962315 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962318 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962321 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962332 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962336 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962339 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962342 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962345 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962351 (+ 6us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962893 (+ 542us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962895 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962897 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960606 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960608 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960609 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960610 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960615 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960620 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960621 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960622 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960623 (+ 1us) write_transaction.cc:95] Timestamp: L: 517 1103 05:10:05.962297 (+ 1674us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962300 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962302 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962304 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962306 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962310 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962314 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962316 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962318 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962320 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962333 (+ 13us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962335 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962338 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962341 (+ 3us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962884 (+ 543us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962886 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962887 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960624 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960626 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960627 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960628 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960632 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960637 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960638 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960639 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960640 (+ 1us) write_transaction.cc:95] Timestamp: L: 518 1103 05:10:05.962302 (+ 1662us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962305 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962307 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962310 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962312 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962314 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962316 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962319 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962326 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962328 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962331 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962333 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962335 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962339 (+ 4us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962876 (+ 537us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962878 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962879 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960642 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960643 (+ 1us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960644 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960645 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960650 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960654 (+ 4us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960655 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960656 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960658 (+ 2us) write_transaction.cc:95] Timestamp: L: 519 1103 05:10:05.962341 (+ 1683us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962344 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962346 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962348 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962350 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962363 (+ 13us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962365 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962367 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962370 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962373 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962375 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962377 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:05.962379 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962383 (+ 4us) log.cc:406] Serialized 86 byte log entry 1103 05:10:05.962903 (+ 520us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:05.962906 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:05.962907 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:05.960659 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:05.960661 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:05.960662 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:05.960663 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:05.960667 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:05.960672 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:05.960673 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:05.960674 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:05.960675 (+ 1us) write_transaction.cc:95] Timestamp: L: 520 1103 05:10:05.962344 (+ 1669us) write_transaction.cc:102] APPLY: Starting 1103 05:10:05.962348 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962363 (+ 15us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962366 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962368 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962371 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962373 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962375 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962379 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:05.962381 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,k W1103 05:10:06.019652 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.580-1.581] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:06.020727 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 28818.9 usec... I1103 05:10:06.030316 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 44345.7 usec... I1103 05:10:06.031142 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 3972.95 usec... I1103 05:10:06.040719 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 10213.7 usec... I1103 05:10:06.051585 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 7236.89 usec... I1103 05:10:06.055047 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 8686.83 usec... I1103 05:10:06.065480 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 1774.21 usec... I1103 05:10:06.075124 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15395.4 usec... I1103 05:10:06.090736 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 10490.6 usec... I1103 05:10:06.101860 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 7286.44 usec... I1103 05:10:06.128751 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 15316.2 usec... I1103 05:10:06.137418 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 14819.7 usec... I1103 05:10:06.144876 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 33653 usec... I1103 05:10:06.145758 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12139.3 usec... I1103 05:10:06.153333 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 49126.8 usec... I1103 05:10:06.178555 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 44946.7 usec... I1103 05:10:06.178997 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 18441.6 usec... W1103 05:10:06.202716 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.750-1.751] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:06.203457 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.774-1.774] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:06.230016 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 1694.18 usec... W1103 05:10:06.246094 31061 thread.cc:320] Waited for 1000ms trying to join with ts-test0 (tid 8513) I1103 05:10:06.269238 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 3275.49 usec... I1103 05:10:06.272900 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 52610.5 usec... I1103 05:10:06.273071 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 65246.8 usec... I1103 05:10:06.302371 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 27647.9 usec... I1103 05:10:06.325767 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 46322.7 usec... I1103 05:10:06.330657 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 43453.8 usec... I1103 05:10:06.338558 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 60548.9 usec... W1103 05:10:06.372669 7783 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 238) took 46ms (client timeout 50). W1103 05:10:06.372979 7783 inbound_call.cc:140] Trace: 1103 05:10:06.325765 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:06.372405 (+ 46640us) service_pool.cc:152] Handling call 1103 05:10:06.372449 (+ 44us) raft_consensus.cc:579] Updating replica for 7 ops 1103 05:10:06.372452 (+ 3us) raft_consensus.cc:614] Triggering prepare for 7 ops 1103 05:10:06.372556 (+ 104us) log.cc:406] Serialized 3532 byte log entry 1103 05:10:06.372564 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 960 1103 05:10:06.372608 (+ 44us) raft_consensus.cc:682] Updating last received op as term: 1 index: 964 1103 05:10:06.372610 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:06.372642 (+ 32us) raft_consensus.cc:697] finished 1103 05:10:06.372643 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:06.372645 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:06.372651 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:06.372488 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.372495 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.372496 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.372497 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.372505 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.372515 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.372517 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.372518 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:06.372521 (+ 3us) write_transaction.cc:95] Timestamp: L: 1732 1103 05:10:06.372610 (+ 89us) write_transaction.cc:102] APPLY: Starting 1103 05:10:06.372617 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372620 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372623 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372627 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372633 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372636 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372639 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372642 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372644 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372647 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372650 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:06.372656 (+ 6us) write_transaction.cc:228] Released schema lock 1103 05:10:06.372663 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:06.372524 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.372526 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.372527 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.372528 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.372534 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.372541 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.372542 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.372543 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:06.372544 (+ 1us) write_transaction.cc:95] Timestamp: L: 1733 1103 05:10:06.372615 (+ 71us) write_transaction.cc:102] APPLY: Starting 1103 05:10:06.372621 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372624 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372627 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372630 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372655 (+ 25us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372659 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372662 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372665 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372668 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372671 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372674 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:06.372678 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:06.372685 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:06.372546 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.372549 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.372549 (+ 0us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.372550 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.372556 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.372563 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.372564 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.372565 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:06.372566 (+ 1us) write_transaction.cc:95] Timestamp: L: 1734 1103 05:10:06.372657 (+ 91us) write_transaction.cc:102] APPLY: Starting 1103 05:10:06.372661 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372663 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372666 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372669 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372697 (+ 28us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372701 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372705 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372707 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372710 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372713 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:06.372716 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:06.372719 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:06.372726 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:06.372568 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.372570 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.372571 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.372572 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.372577 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.372583 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.372584 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.372586 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:06.372587 (+ 1us) write_transaction.cc:95] Timestamp: L: 1735 Related trace: 1103 05:10:06.372588 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.372590 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.372591 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.372592 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.372598 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.372608 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.372610 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.372611 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:06.372614 (+ 3us) write_transaction.cc:95] Timestamp: L: 1736 Related trace: 1103 05:10:06.372616 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.372619 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.372621 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.372622 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.372629 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.372639 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.372641 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.372642 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:06.372645 (+ 3us) write_transaction.cc:95] Timestamp: L: 1737 Related trace: 1103 05:10:06.372647 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.372650 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.372651 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.372652 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.372659 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.372669 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.372670 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.372672 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:06.372675 (+ 3us) write_transaction.cc:95] Timestamp: L: 1738 I1103 05:10:06.374636 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 32616.2 usec... I1103 05:10:06.397765 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15206.2 usec... W1103 05:10:06.399339 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.958-1.964] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:06.399855 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 17923.9 usec... I1103 05:10:06.407488 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 2171.43 usec... I1103 05:10:06.410369 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 6551.7 usec... I1103 05:10:06.413722 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 26647.9 usec... W1103 05:10:06.399344 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.965-1.966] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:06.418009 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 9406.28 usec... I1103 05:10:06.427650 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 9600.67 usec... I1103 05:10:06.437046 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 36506.7 usec... I1103 05:10:06.437949 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 1760.91 usec... I1103 05:10:06.441088 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 1701.11 usec... I1103 05:10:06.442945 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 14502.5 usec... I1103 05:10:06.457738 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 21449.7 usec... I1103 05:10:06.459867 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 2577.71 usec... I1103 05:10:06.462690 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 42713.8 usec... I1103 05:10:06.479888 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 24072.9 usec... W1103 05:10:06.504302 7779 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 239) took 63ms (client timeout 50). W1103 05:10:06.504407 7779 inbound_call.cc:140] Trace: 1103 05:10:06.441085 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:06.442930 (+ 1845us) service_pool.cc:152] Handling call 1103 05:10:06.479384 (+ 36454us) raft_consensus.cc:579] Updating replica for 10 ops 1103 05:10:06.504166 (+ 24782us) raft_consensus.cc:614] Triggering prepare for 3 ops 1103 05:10:06.504220 (+ 54us) log.cc:406] Serialized 1508 byte log entry 1103 05:10:06.504229 (+ 9us) raft_consensus.cc:669] Marking committed up to term: 1 index: 964 1103 05:10:06.504273 (+ 44us) raft_consensus.cc:682] Updating last received op as term: 1 index: 967 1103 05:10:06.504275 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:06.504276 (+ 1us) raft_consensus.cc:697] finished 1103 05:10:06.504278 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:06.504279 (+ 1us) raft_consensus.cc:443] Updating watermarks 1103 05:10:06.504285 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:06.504201 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.504208 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.504209 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.504210 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.504219 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.504229 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.504231 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.504233 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:06.504236 (+ 3us) write_transaction.cc:95] Timestamp: L: 1743 Related trace: 1103 05:10:06.504239 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.504241 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.504242 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.504243 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.504249 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.504258 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.504259 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.504260 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:06.504262 (+ 2us) write_transaction.cc:95] Timestamp: L: 1744 Related trace: 1103 05:10:06.504263 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.504265 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.504266 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.504267 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.504275 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.504285 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.504286 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.504288 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:06.504290 (+ 2us) write_transaction.cc:95] Timestamp: L: 1745 I1103 05:10:06.504891 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12913.2 usec... W1103 05:10:06.505647 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.965-1.967] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. W1103 05:10:06.505664 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.958-1.967] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:06.518301 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 54491.3 usec... I1103 05:10:06.556493 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16460.2 usec... I1103 05:10:06.573097 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 53894.5 usec... I1103 05:10:06.597321 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 61622.5 usec... I1103 05:10:06.602632 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 15383.3 usec... I1103 05:10:06.627238 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 23671 usec... I1103 05:10:06.639145 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 31598.5 usec... I1103 05:10:06.651136 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 6026.39 usec... I1103 05:10:06.659437 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 22911.8 usec... I1103 05:10:06.688256 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15681.9 usec... I1103 05:10:06.698911 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 17839.2 usec... I1103 05:10:06.717232 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 10093.3 usec... I1103 05:10:06.727565 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 5605.38 usec... I1103 05:10:06.733403 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 9886.96 usec... I1103 05:10:06.738864 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 39280.1 usec... I1103 05:10:06.782232 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 17161.8 usec... I1103 05:10:06.799619 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 18712.9 usec... I1103 05:10:06.838217 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 25811.5 usec... I1103 05:10:06.846946 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 770.594 usec... I1103 05:10:06.864588 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 8065.11 usec... I1103 05:10:06.901721 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 30727.5 usec... I1103 05:10:06.932674 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 13630.7 usec... I1103 05:10:06.946527 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 19325.7 usec... I1103 05:10:06.953588 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 60868 usec... I1103 05:10:06.960975 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 7117.19 usec... I1103 05:10:06.966123 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 2553.97 usec... I1103 05:10:06.968710 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 49608.1 usec... I1103 05:10:06.969094 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 10196.8 usec... I1103 05:10:06.979707 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 38040 usec... W1103 05:10:07.015130 7580 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 291) took 61ms (client timeout 50). W1103 05:10:07.015224 7580 inbound_call.cc:140] Trace: 1103 05:10:06.953527 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:06.953554 (+ 27us) service_pool.cc:152] Handling call 1103 05:10:06.953570 (+ 16us) raft_consensus.cc:579] Updating replica for 2 ops 1103 05:10:06.953572 (+ 2us) raft_consensus.cc:614] Triggering prepare for 2 ops 1103 05:10:06.953616 (+ 44us) log.cc:406] Serialized 1012 byte log entry 1103 05:10:06.953624 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1006 1103 05:10:06.953636 (+ 12us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1008 1103 05:10:06.953638 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:07.015102 (+ 61464us) raft_consensus.cc:697] finished 1103 05:10:07.015105 (+ 3us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:07.015107 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:07.015112 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:06.953610 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.953616 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.953617 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.953618 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.953625 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.953634 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.953636 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.953638 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:06.953641 (+ 3us) write_transaction.cc:95] Timestamp: L: 1740 Related trace: 1103 05:10:06.953644 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:06.953648 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:06.953649 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:06.953650 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:06.953656 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:06.953662 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:06.953664 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:06.953665 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:06.953668 (+ 3us) write_transaction.cc:95] Timestamp: L: 1741 W1103 05:10:07.018582 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1007-1.1008] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:07.018620 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.958-1.972] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 3 times. I1103 05:10:07.037232 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 42518.1 usec... I1103 05:10:07.064990 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 11283.5 usec... I1103 05:10:07.078439 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 34281.4 usec... I1103 05:10:07.121747 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 23751 usec... I1103 05:10:07.132004 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 45161.7 usec... I1103 05:10:07.178095 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 23779.7 usec... I1103 05:10:07.180248 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 15239.6 usec... I1103 05:10:07.199702 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 13388 usec... W1103 05:10:07.246279 31061 thread.cc:320] Waited for 2000ms trying to join with ts-test0 (tid 8513) I1103 05:10:07.253981 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 21540.4 usec... I1103 05:10:07.276157 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 545.368 usec... I1103 05:10:07.311560 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 48322.8 usec... I1103 05:10:07.318164 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 48623.7 usec... I1103 05:10:07.352291 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 41990.1 usec... I1103 05:10:07.367246 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 38885.4 usec... I1103 05:10:07.394917 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 20884.4 usec... I1103 05:10:07.401669 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 20149.5 usec... I1103 05:10:07.422212 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 23450.7 usec... I1103 05:10:07.422528 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 13487.4 usec... I1103 05:10:07.445889 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 23965.3 usec... I1103 05:10:07.463690 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 32268.2 usec... I1103 05:10:07.470087 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 3232.22 usec... W1103 05:10:07.491610 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.958-1.1014] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 4 times. I1103 05:10:07.496346 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 2306.84 usec... I1103 05:10:07.498880 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 573.744 usec... W1103 05:10:07.504102 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1007-1.1014] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:07.527230 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 25156.5 usec... I1103 05:10:07.533123 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 17415.6 usec... I1103 05:10:07.550748 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 30505.3 usec... I1103 05:10:07.552464 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 9388.32 usec... I1103 05:10:07.562079 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 23718.8 usec... I1103 05:10:07.575191 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 47630.4 usec... I1103 05:10:07.599042 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 41231.3 usec... I1103 05:10:07.609782 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 51240.9 usec... I1103 05:10:07.623034 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 35395.4 usec... I1103 05:10:07.684075 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 25411.6 usec... I1103 05:10:07.694277 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 17879.8 usec... I1103 05:10:07.712843 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 14715.6 usec... I1103 05:10:07.717695 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 7862.09 usec... I1103 05:10:07.726022 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 17600.6 usec... I1103 05:10:07.744072 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 17589.5 usec... I1103 05:10:07.767421 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15477.7 usec... I1103 05:10:07.773118 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 9535.77 usec... I1103 05:10:07.783288 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 10917.7 usec... I1103 05:10:07.794726 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 81602.3 usec... W1103 05:10:07.851933 8084 log.cc:507] Time spent Append to log took a long time: real 0.410s user 0.000s sys 0.000s W1103 05:10:07.852021 7786 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 241) took 410ms (client timeout 50). W1103 05:10:07.853306 7786 inbound_call.cc:140] Trace: 1103 05:10:07.441200 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:07.441221 (+ 21us) service_pool.cc:152] Handling call 1103 05:10:07.441392 (+ 171us) raft_consensus.cc:579] Updating replica for 57 ops 1103 05:10:07.441397 (+ 5us) raft_consensus.cc:614] Triggering prepare for 42 ops 1103 05:10:07.441833 (+ 436us) log.cc:406] Serialized 21152 byte log entry 1103 05:10:07.441841 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1006 1103 05:10:07.442435 (+ 594us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1014 1103 05:10:07.442440 (+ 5us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:07.851980 (+409540us) raft_consensus.cc:697] finished 1103 05:10:07.851983 (+ 3us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:07.851986 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:10:07.851994 (+ 8us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:07.441430 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441437 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441438 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441439 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441448 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441458 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441459 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441460 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441464 (+ 4us) write_transaction.cc:95] Timestamp: L: 1754 1103 05:10:07.442458 (+ 994us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442462 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442465 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442467 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442470 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442475 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442477 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442480 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442482 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442485 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442487 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442489 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442492 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442496 (+ 4us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441466 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441469 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441470 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441471 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441476 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441486 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441487 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441489 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:07.441490 (+ 1us) write_transaction.cc:95] Timestamp: L: 1755 1103 05:10:07.442457 (+ 967us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442462 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442465 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442468 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442472 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442478 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442481 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442484 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442487 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442491 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442494 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442497 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442501 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442506 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441492 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441494 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441495 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441496 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441501 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441508 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441509 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441510 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441511 (+ 1us) write_transaction.cc:95] Timestamp: L: 1756 1103 05:10:07.442465 (+ 954us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442470 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442473 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442475 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442478 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442483 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442485 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442487 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442489 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442492 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442494 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442496 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442500 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442506 (+ 6us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441513 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441516 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441517 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441517 (+ 0us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441522 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441529 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441530 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441531 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441532 (+ 1us) write_transaction.cc:95] Timestamp: L: 1757 1103 05:10:07.442500 (+ 968us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442503 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442505 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442507 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442513 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442516 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442519 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442521 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442524 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442526 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442528 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442530 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442533 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442536 (+ 3us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441534 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441536 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441537 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441538 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441543 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441549 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441550 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441551 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441552 (+ 1us) write_transaction.cc:95] Timestamp: L: 1758 1103 05:10:07.442514 (+ 962us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442519 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442522 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442525 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442528 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442533 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442537 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442540 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442543 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442547 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442550 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442552 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442557 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442562 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441554 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441556 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441557 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441558 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441563 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441570 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441571 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441572 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441574 (+ 2us) write_transaction.cc:95] Timestamp: L: 1759 1103 05:10:07.442524 (+ 950us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442529 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442531 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442533 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442535 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442538 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442541 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442544 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442547 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442549 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442551 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442553 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442557 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442560 (+ 3us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441575 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441577 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441578 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441579 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441584 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441590 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441591 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441592 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441593 (+ 1us) write_transaction.cc:95] Timestamp: L: 1760 1103 05:10:07.442533 (+ 940us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442538 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442541 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442544 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442547 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442553 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442556 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442559 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442562 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442573 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442576 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442578 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442583 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442589 (+ 6us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441595 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441597 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441598 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441599 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441604 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441609 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441610 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441612 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:07.441613 (+ 1us) write_transaction.cc:95] Timestamp: L: 1761 1103 05:10:07.442539 (+ 926us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442542 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442544 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442546 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442550 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442553 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442557 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442559 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442561 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442563 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442571 (+ 8us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442573 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442575 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442579 (+ 4us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441615 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441618 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441620 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441621 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441628 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441637 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441638 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441640 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:07.441642 (+ 2us) write_transaction.cc:95] Timestamp: L: 1762 1103 05:10:07.442563 (+ 921us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442573 (+ 10us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442575 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442578 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442582 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442585 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442587 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442590 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442592 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442594 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442597 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442600 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442602 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442606 (+ 4us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441645 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441649 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441649 (+ 0us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441650 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441656 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441667 (+ 11us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441668 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441669 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441673 (+ 4us) write_transaction.cc:95] Timestamp: L: 1763 1103 05:10:07.442567 (+ 894us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442572 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442575 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442578 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442585 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442589 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442593 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442597 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442600 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442603 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442606 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442609 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442613 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442620 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441678 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441680 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441681 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441682 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441687 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441693 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441694 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441695 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441696 (+ 1us) write_transaction.cc:95] Timestamp: L: 1764 1103 05:10:07.442582 (+ 886us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442586 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442589 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442591 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442593 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442596 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442599 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442602 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442604 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442607 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442614 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442617 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:07.442620 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:07.442624 (+ 4us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:07.441699 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.441701 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.441702 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.441703 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.441707 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.441713 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.441714 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.441715 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.441716 (+ 1us) write_transaction.cc:95] Timestamp: L: 1765 1103 05:10:07.442594 (+ 878us) write_transaction.cc:102] APPLY: Starting 1103 05:10:07.442598 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442601 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442604 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442607 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442618 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442622 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442625 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442629 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:07.442632 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas I1103 05:10:07.869410 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 33269.8 usec... W1103 05:10:07.876755 8085 log.cc:507] Time spent Append to log took a long time: real 0.403s user 0.000s sys 0.000s W1103 05:10:07.876842 7583 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 292) took 403ms (client timeout 50). I1103 05:10:07.877058 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 7220.76 usec... I1103 05:10:07.884505 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 23016.5 usec... I1103 05:10:07.901490 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 37680 usec... I1103 05:10:07.902926 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 28021.1 usec... W1103 05:10:07.908092 7583 inbound_call.cc:140] Trace: 1103 05:10:07.473542 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:07.473560 (+ 18us) service_pool.cc:152] Handling call 1103 05:10:07.473615 (+ 55us) raft_consensus.cc:579] Updating replica for 8 ops 1103 05:10:07.473618 (+ 3us) raft_consensus.cc:614] Triggering prepare for 6 ops 1103 05:10:07.473858 (+ 240us) log.cc:406] Serialized 3016 byte log entry 1103 05:10:07.473868 (+ 10us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1006 1103 05:10:07.473871 (+ 3us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1014 1103 05:10:07.473872 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:07.876813 (+402941us) raft_consensus.cc:697] finished 1103 05:10:07.876815 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:07.876818 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:10:07.876826 (+ 8us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:07.473785 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.473793 (+ 8us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.473794 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.473795 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.473807 (+ 12us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.473818 (+ 11us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.473819 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.473821 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:07.473824 (+ 3us) write_transaction.cc:95] Timestamp: L: 1743 Related trace: 1103 05:10:07.473827 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.473830 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.473831 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.473832 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.473837 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.473845 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.473846 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.473847 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.473849 (+ 2us) write_transaction.cc:95] Timestamp: L: 1744 Related trace: 1103 05:10:07.473850 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.473892 (+ 42us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.473893 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.473894 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.473900 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.473918 (+ 18us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.473919 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.473921 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:07.473923 (+ 2us) write_transaction.cc:95] Timestamp: L: 1745 Related trace: 1103 05:10:07.473926 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.473928 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.473929 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.473930 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.473935 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.473941 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.473942 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.473944 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:07.473945 (+ 1us) write_transaction.cc:95] Timestamp: L: 1746 Related trace: 1103 05:10:07.473947 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.473950 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.473951 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.473952 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.473956 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.473963 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.473964 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.473965 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.473966 (+ 1us) write_transaction.cc:95] Timestamp: L: 1747 Related trace: 1103 05:10:07.473968 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:07.473970 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:07.473971 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:07.473972 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:07.473977 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:07.473984 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:07.473985 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:07.473986 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:07.473987 (+ 1us) write_transaction.cc:95] Timestamp: L: 1748 I1103 05:10:07.908108 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 79255.1 usec... I1103 05:10:07.945806 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 41173.6 usec... I1103 05:10:07.963174 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12092.5 usec... I1103 05:10:07.975663 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 2776.82 usec... I1103 05:10:07.988134 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 45759.1 usec... I1103 05:10:08.034123 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 22206.5 usec... I1103 05:10:08.046416 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 52447.1 usec... I1103 05:10:08.057291 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15382.3 usec... I1103 05:10:08.073627 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 21916 usec... I1103 05:10:08.095767 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 69806.1 usec... W1103 05:10:08.099148 8093 log.cc:507] Time spent Append to log took a long time: real 0.157s user 0.000s sys 0.002s I1103 05:10:08.106371 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 35423.6 usec... I1103 05:10:08.147004 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 22971.9 usec... W1103 05:10:08.166352 8085 log.cc:507] Time spent Append to log took a long time: real 0.066s user 0.000s sys 0.000s W1103 05:10:08.166416 7587 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 294) took 66ms (client timeout 50). W1103 05:10:08.166466 7587 inbound_call.cc:140] Trace: 1103 05:10:08.100014 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:08.100037 (+ 23us) service_pool.cc:152] Handling call 1103 05:10:08.100056 (+ 19us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:08.100058 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:08.100213 (+ 155us) log.cc:406] Serialized 506 byte log entry 1103 05:10:08.100225 (+ 12us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1014 1103 05:10:08.100229 (+ 4us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1015 1103 05:10:08.100231 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:08.166392 (+ 66161us) raft_consensus.cc:697] finished 1103 05:10:08.166394 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:08.166396 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:08.166402 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:08.100209 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:08.100224 (+ 15us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:08.100226 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:08.100227 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:08.100237 (+ 10us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:08.100250 (+ 13us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:08.100251 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:08.100253 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:08.100256 (+ 3us) write_transaction.cc:95] Timestamp: L: 1756 I1103 05:10:08.166744 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 127.99 usec... I1103 05:10:08.167089 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 21135.1 usec... W1103 05:10:08.170182 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1015-1.1015] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:08.170235 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1015-1.1015] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:08.189409 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 16016.9 usec... I1103 05:10:08.203145 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 19246 usec... W1103 05:10:08.206320 7787 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 243) took 40ms (client timeout 50). W1103 05:10:08.206388 7787 inbound_call.cc:140] Trace: 1103 05:10:08.166354 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:08.166367 (+ 13us) service_pool.cc:152] Handling call 1103 05:10:08.166387 (+ 20us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:08.166389 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:08.166490 (+ 101us) log.cc:406] Serialized 506 byte log entry 1103 05:10:08.166501 (+ 11us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1014 1103 05:10:08.206291 (+ 39790us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1015 1103 05:10:08.206297 (+ 6us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:08.206298 (+ 1us) raft_consensus.cc:697] finished 1103 05:10:08.206299 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:08.206300 (+ 1us) raft_consensus.cc:443] Updating watermarks 1103 05:10:08.206305 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:08.188685 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:08.188696 (+ 11us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:08.188698 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:08.188699 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:08.188713 (+ 14us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:08.188727 (+ 14us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:08.188728 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:08.188730 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:08.188734 (+ 4us) write_transaction.cc:95] Timestamp: L: 1802 I1103 05:10:08.243625 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 19306.7 usec... W1103 05:10:08.246419 31061 thread.cc:320] Waited for 3000ms trying to join with ts-test0 (tid 8513) I1103 05:10:08.252845 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 13553.2 usec... I1103 05:10:08.264287 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 18255.9 usec... I1103 05:10:08.266615 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 13085.1 usec... I1103 05:10:08.279916 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 47051.9 usec... I1103 05:10:08.283020 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 17602.8 usec... I1103 05:10:08.301012 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 18495.5 usec... I1103 05:10:08.316377 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 32247.9 usec... I1103 05:10:08.319763 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 19285.2 usec... I1103 05:10:08.327210 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 65582.1 usec... I1103 05:10:08.352393 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 6342.67 usec... I1103 05:10:08.354266 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 5002.26 usec... I1103 05:10:08.360229 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 25936.3 usec... I1103 05:10:08.386559 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 4493.01 usec... I1103 05:10:08.393020 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 17424.3 usec... I1103 05:10:08.409766 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 7914.59 usec... I1103 05:10:08.410675 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16192.5 usec... I1103 05:10:08.413486 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 17583.5 usec... I1103 05:10:08.418771 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 11164.1 usec... I1103 05:10:08.427098 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 40974.8 usec... I1103 05:10:08.431704 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 36518.1 usec... I1103 05:10:08.469009 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 18297.7 usec... I1103 05:10:08.472637 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 60216.7 usec... I1103 05:10:08.489182 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 35762.6 usec... I1103 05:10:08.512997 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 4552.16 usec... I1103 05:10:08.533090 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 380.926 usec... I1103 05:10:08.533808 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 5430.76 usec... I1103 05:10:08.539943 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 75099.2 usec... I1103 05:10:08.575326 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 30063.7 usec... I1103 05:10:08.593653 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 64962.6 usec... I1103 05:10:08.615296 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 28133.2 usec... W1103 05:10:08.655760 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1015-1.1022] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:08.668059 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 2426.82 usec... I1103 05:10:08.670732 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 73183.5 usec... I1103 05:10:08.679201 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 48536.8 usec... W1103 05:10:08.727957 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1122-1.1122] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:08.727977 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 4107.89 usec... W1103 05:10:08.732347 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1172-1.1172] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:08.744171 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 7394.19 usec... I1103 05:10:08.745307 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 74694 usec... W1103 05:10:08.751835 8085 log.cc:507] Time spent Append to log took a long time: real 0.084s user 0.000s sys 0.000s W1103 05:10:08.751968 7580 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 331) took 84ms (client timeout 50). W1103 05:10:08.752017 7580 inbound_call.cc:140] Trace: 1103 05:10:08.667549 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:08.667564 (+ 15us) service_pool.cc:152] Handling call 1103 05:10:08.667572 (+ 8us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:08.667574 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:08.667599 (+ 25us) log.cc:406] Serialized 506 byte log entry 1103 05:10:08.667606 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1121 1103 05:10:08.667653 (+ 47us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1122 1103 05:10:08.667655 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:08.751949 (+ 84294us) raft_consensus.cc:697] finished 1103 05:10:08.751950 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:08.751953 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:10:08.751958 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:08.667608 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:08.667613 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:08.667614 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:08.667615 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:08.667621 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:08.667629 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:08.667631 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:08.667633 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:08.667636 (+ 3us) write_transaction.cc:95] Timestamp: L: 1943 I1103 05:10:08.767535 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 2925.93 usec... W1103 05:10:08.795418 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1181-1.1181] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:08.811393 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 14493.6 usec... I1103 05:10:08.820266 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 32531.9 usec... I1103 05:10:08.826179 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 24284.8 usec... W1103 05:10:08.850798 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1181-1.1281] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:08.853047 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 28950.5 usec... W1103 05:10:08.883056 7780 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 271) took 137ms (client timeout 50). W1103 05:10:08.883158 7780 inbound_call.cc:140] Trace: 1103 05:10:08.745281 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:08.745302 (+ 21us) service_pool.cc:152] Handling call 1103 05:10:08.745317 (+ 15us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:08.745319 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:08.745353 (+ 34us) log.cc:406] Serialized 506 byte log entry 1103 05:10:08.883010 (+137657us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1180 1103 05:10:08.883028 (+ 18us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1181 1103 05:10:08.883030 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:08.883031 (+ 1us) raft_consensus.cc:697] finished 1103 05:10:08.883033 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:08.883035 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:08.883042 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:08.882311 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:08.882319 (+ 8us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:08.882321 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:08.882322 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:08.882334 (+ 12us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:08.882350 (+ 16us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:08.882351 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:08.882353 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:08.882357 (+ 4us) write_transaction.cc:95] Timestamp: L: 2056 I1103 05:10:08.888070 7779 raft_consensus.cc:608] T 69ce28a55ab34dafb931a4677a97f8ae P e95cd7e351fc4e28bac7f7977928f571 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 1 index: 1329. Preceding OpId from leader: term: 1 index: 1281. I1103 05:10:08.893688 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15316.3 usec... I1103 05:10:08.945171 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 30556 usec... I1103 05:10:08.951774 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 40572.6 usec... I1103 05:10:08.976027 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15079.3 usec... I1103 05:10:08.989112 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 31563.1 usec... I1103 05:10:08.991423 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 41502.3 usec... W1103 05:10:08.992746 7579 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 522) took 41ms (client timeout 50). W1103 05:10:08.992877 7579 inbound_call.cc:140] Trace: 1103 05:10:08.951192 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:08.951225 (+ 33us) service_pool.cc:152] Handling call 1103 05:10:08.951240 (+ 15us) raft_consensus.cc:579] Updating replica for 2 ops 1103 05:10:08.951242 (+ 2us) raft_consensus.cc:614] Triggering prepare for 2 ops 1103 05:10:08.951291 (+ 49us) log.cc:406] Serialized 1012 byte log entry 1103 05:10:08.951298 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1631 1103 05:10:08.951316 (+ 18us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1633 1103 05:10:08.951318 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:08.992711 (+ 41393us) raft_consensus.cc:697] finished 1103 05:10:08.992713 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:08.992715 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:08.992729 (+ 14us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:08.951276 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:08.951282 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:08.951283 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:08.951284 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:08.951292 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:08.951301 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:08.951302 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:08.951304 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:08.951307 (+ 3us) write_transaction.cc:95] Timestamp: L: 2853 Related trace: 1103 05:10:08.951310 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:08.951314 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:08.951315 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:08.951316 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:08.951323 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:08.951331 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:08.951332 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:08.951334 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:08.951336 (+ 2us) write_transaction.cc:95] Timestamp: L: 2854 I1103 05:10:08.992980 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 57056.3 usec... W1103 05:10:09.020902 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1632-1.1633] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:09.020964 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1608-1.1610] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:09.033361 7783 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 300) took 41ms (client timeout 50). W1103 05:10:09.033462 7783 inbound_call.cc:140] Trace: 1103 05:10:08.991417 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:09.033119 (+ 41702us) service_pool.cc:152] Handling call 1103 05:10:09.033146 (+ 27us) raft_consensus.cc:579] Updating replica for 3 ops 1103 05:10:09.033148 (+ 2us) raft_consensus.cc:614] Triggering prepare for 3 ops 1103 05:10:09.033192 (+ 44us) log.cc:406] Serialized 1518 byte log entry 1103 05:10:09.033199 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1607 1103 05:10:09.033208 (+ 9us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1610 1103 05:10:09.033209 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:09.033337 (+ 128us) raft_consensus.cc:697] finished 1103 05:10:09.033338 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:09.033340 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:09.033345 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:09.033182 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.033191 (+ 9us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.033192 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.033194 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.033204 (+ 10us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.033230 (+ 26us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.033231 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.033234 (+ 3us) write_transaction.cc:93] Start() 1103 05:10:09.033237 (+ 3us) write_transaction.cc:95] Timestamp: L: 2821 Related trace: 1103 05:10:09.033241 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.033245 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.033247 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.033248 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.033256 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.033272 (+ 16us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.033273 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.033275 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.033278 (+ 3us) write_transaction.cc:95] Timestamp: L: 2822 Related trace: 1103 05:10:09.033281 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.033284 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.033286 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.033287 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.033297 (+ 10us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.033307 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.033308 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.033310 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.033313 (+ 3us) write_transaction.cc:95] Timestamp: L: 2824 I1103 05:10:09.069542 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 21126 usec... I1103 05:10:09.073634 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 14879.1 usec... I1103 05:10:09.088734 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 63945.5 usec... I1103 05:10:09.091012 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 1932.91 usec... I1103 05:10:09.093808 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 27852.2 usec... I1103 05:10:09.109488 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 26712.9 usec... I1103 05:10:09.122403 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 33427 usec... I1103 05:10:09.156296 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 45041.9 usec... I1103 05:10:09.172124 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 49493.4 usec... I1103 05:10:09.200328 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 56280.1 usec... I1103 05:10:09.220448 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 49555.2 usec... W1103 05:10:09.246556 31061 thread.cc:320] Waited for 4000ms trying to join with ts-test0 (tid 8513) I1103 05:10:09.246853 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 18225.4 usec... I1103 05:10:09.256872 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 32011.3 usec... I1103 05:10:09.278727 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 33708.3 usec... I1103 05:10:09.289276 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 2309.95 usec... I1103 05:10:09.310029 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 44042.1 usec... I1103 05:10:09.312671 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 4926.05 usec... I1103 05:10:09.317850 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 6622.64 usec... I1103 05:10:09.324699 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 37673.7 usec... I1103 05:10:09.355512 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 7956.69 usec... I1103 05:10:09.363924 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 11745.1 usec... I1103 05:10:09.379907 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 37227.3 usec... I1103 05:10:09.380091 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 10504.3 usec... I1103 05:10:09.391149 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 49834.2 usec... I1103 05:10:09.398311 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 18275.6 usec... I1103 05:10:09.417384 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12840.8 usec... I1103 05:10:09.417676 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 72098.9 usec... I1103 05:10:09.441995 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 37322.1 usec... I1103 05:10:09.444849 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 31867.9 usec... I1103 05:10:09.477632 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 29433.3 usec... I1103 05:10:09.479542 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 2100.39 usec... I1103 05:10:09.490345 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 21945.3 usec... I1103 05:10:09.519335 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 261.129 usec... I1103 05:10:09.520448 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 36591.3 usec... I1103 05:10:09.533387 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 31892.5 usec... I1103 05:10:09.557381 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 31817.5 usec... W1103 05:10:09.570581 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1641-1.1648] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:09.578479 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 59524.4 usec... I1103 05:10:09.582471 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 17935.7 usec... I1103 05:10:09.589431 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 10911.8 usec... I1103 05:10:09.600652 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 18462.4 usec... W1103 05:10:09.601152 7583 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 525) took 81ms (client timeout 50). W1103 05:10:09.601503 7583 inbound_call.cc:140] Trace: 1103 05:10:09.520101 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:09.520125 (+ 24us) service_pool.cc:152] Handling call 1103 05:10:09.600564 (+ 80439us) raft_consensus.cc:579] Updating replica for 8 ops 1103 05:10:09.600569 (+ 5us) raft_consensus.cc:614] Triggering prepare for 8 ops 1103 05:10:09.600690 (+ 121us) log.cc:406] Serialized 4028 byte log entry 1103 05:10:09.600699 (+ 9us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1644 1103 05:10:09.600715 (+ 16us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1648 1103 05:10:09.600720 (+ 5us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:09.601125 (+ 405us) raft_consensus.cc:697] finished 1103 05:10:09.601126 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:09.601128 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:09.601138 (+ 10us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:09.600783 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.600791 (+ 8us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.600792 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.600793 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.600805 (+ 12us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.600820 (+ 15us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.600821 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.600823 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.600826 (+ 3us) write_transaction.cc:95] Timestamp: L: 2866 1103 05:10:09.600904 (+ 78us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.600908 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600911 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600913 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600916 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600922 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600925 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600927 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600929 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600931 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600933 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.600935 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.600940 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:09.600949 (+ 9us) log.cc:406] Serialized 86 byte log entry 1103 05:10:09.600982 (+ 33us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:09.600984 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:09.600985 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:09.600834 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.600837 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.600838 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.600839 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.600859 (+ 20us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.600868 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.600870 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.600872 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.600874 (+ 2us) write_transaction.cc:95] Timestamp: L: 2868 1103 05:10:09.601003 (+ 129us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.601007 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601009 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601011 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601013 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601017 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601019 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601022 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601023 (+ 1us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601025 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601037 (+ 12us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601039 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.601042 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:09.601050 (+ 8us) log.cc:406] Serialized 86 byte log entry 1103 05:10:09.601142 (+ 92us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:09.601144 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:09.601146 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:09.600881 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.600884 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.600886 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.600887 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.600894 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.600905 (+ 11us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.600906 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.600908 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.600910 (+ 2us) write_transaction.cc:95] Timestamp: L: 2870 1103 05:10:09.601053 (+ 143us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.601057 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601060 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601062 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601064 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601070 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601072 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601074 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601077 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601079 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601081 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601083 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.601086 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:09.601088 (+ 2us) log.cc:406] Serialized 86 byte log entry 1103 05:10:09.601151 (+ 63us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:09.601153 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:09.601154 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:09.600916 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.600920 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.600921 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.600922 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.600930 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.600942 (+ 12us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.600943 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.600945 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.600948 (+ 3us) write_transaction.cc:95] Timestamp: L: 2871 1103 05:10:09.601062 (+ 114us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.601068 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601070 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601073 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601075 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601079 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601082 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601084 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601086 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601088 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601090 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.601092 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.601095 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:09.601101 (+ 6us) log.cc:406] Serialized 86 byte log entry 1103 05:10:09.601183 (+ 82us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:09.601185 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:09.601186 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:09.600961 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.600965 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.600966 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.600968 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.600974 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.600982 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.600983 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.600985 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.600989 (+ 4us) write_transaction.cc:95] Timestamp: L: 2873 Related trace: 1103 05:10:09.600990 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.600993 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.600994 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.600995 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.601000 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.601006 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.601007 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.601008 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:09.601010 (+ 2us) write_transaction.cc:95] Timestamp: L: 2874 Related trace: 1103 05:10:09.601011 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.601013 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.601014 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.601015 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.601021 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.601029 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.601030 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.601031 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:09.601033 (+ 2us) write_transaction.cc:95] Timestamp: L: 2875 Related trace: 1103 05:10:09.601034 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.601036 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.601037 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.601038 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.601042 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.601047 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.601048 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.601049 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:09.601050 (+ 1us) write_transaction.cc:95] Timestamp: L: 2876 I1103 05:10:09.633927 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 18923.1 usec... W1103 05:10:09.638185 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1641-1.1709] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:09.638507 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 6530.66 usec... I1103 05:10:09.645333 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 1419.69 usec... I1103 05:10:09.646924 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 13186.8 usec... I1103 05:10:09.654418 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 25538.6 usec... W1103 05:10:09.660500 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1725-1.1725] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:09.669313 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 21060.3 usec... I1103 05:10:09.670222 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 27738.7 usec... I1103 05:10:09.680595 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 54986 usec... I1103 05:10:09.690601 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 967.334 usec... I1103 05:10:09.691787 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 2986.08 usec... I1103 05:10:09.695158 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 77235 usec... I1103 05:10:09.706580 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15029.2 usec... I1103 05:10:09.769419 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 9881.74 usec... W1103 05:10:09.772647 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1641-1.1726] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 3 times. W1103 05:10:09.772650 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1725-1.1727] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:09.773655 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 5123.52 usec... I1103 05:10:09.779261 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 39714.9 usec... I1103 05:10:09.779844 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 35747.8 usec... W1103 05:10:09.858106 8093 log.cc:507] Time spent Append to log took a long time: real 0.198s user 0.000s sys 0.000s I1103 05:10:09.862642 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 21909.1 usec... I1103 05:10:09.894547 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 22618 usec... I1103 05:10:09.939769 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 8214.96 usec... I1103 05:10:09.951524 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 317.105 usec... I1103 05:10:09.952466 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 16656.2 usec... I1103 05:10:09.953404 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 27386.2 usec... W1103 05:10:09.969368 8084 log.cc:507] Time spent Append to log took a long time: real 0.309s user 0.000s sys 0.000s W1103 05:10:09.969425 7783 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 339) took 308ms (client timeout 50). W1103 05:10:09.969470 7783 inbound_call.cc:140] Trace: 1103 05:10:09.660582 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:09.660604 (+ 22us) service_pool.cc:152] Handling call 1103 05:10:09.660623 (+ 19us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:09.660626 (+ 3us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:09.660661 (+ 35us) log.cc:406] Serialized 506 byte log entry 1103 05:10:09.660670 (+ 9us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1724 1103 05:10:09.660699 (+ 29us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1725 1103 05:10:09.660701 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:09.969400 (+308699us) raft_consensus.cc:697] finished 1103 05:10:09.969402 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:09.969404 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:09.969413 (+ 9us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:09.660662 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.660670 (+ 8us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.660672 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.660673 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.660685 (+ 12us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.660705 (+ 20us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.660706 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.660708 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.660712 (+ 4us) write_transaction.cc:95] Timestamp: L: 3039 W1103 05:10:09.969815 7780 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 340) took 308ms (client timeout 50). W1103 05:10:09.969882 7780 inbound_call.cc:140] Trace: 1103 05:10:09.661316 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:09.661337 (+ 21us) service_pool.cc:152] Handling call 1103 05:10:09.969419 (+308082us) raft_consensus.cc:579] Updating replica for 3 ops 1103 05:10:09.969422 (+ 3us) raft_consensus.cc:614] Triggering prepare for 2 ops 1103 05:10:09.969474 (+ 52us) log.cc:406] Serialized 1002 byte log entry 1103 05:10:09.969483 (+ 9us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1724 1103 05:10:09.969485 (+ 2us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1727 1103 05:10:09.969486 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:09.969564 (+ 78us) raft_consensus.cc:697] finished 1103 05:10:09.969566 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:09.969567 (+ 1us) raft_consensus.cc:443] Updating watermarks 1103 05:10:09.969802 (+ 235us) spinlock_profiling.cc:37] Waited 308067960.62136704ns on lock 0x3cee630. stack: 00000000010fac13 00000000011260b1 000000000083ad42 000000000069761c 000000000089f881 000000000112972f 0000003b9a407851 0000003b9a0e894d 1103 05:10:09.969807 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:09.969495 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.969500 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.969501 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.969502 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.969510 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.969519 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.969520 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.969522 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.969525 (+ 3us) write_transaction.cc:95] Timestamp: L: 3042 Related trace: 1103 05:10:09.969527 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.969529 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.969530 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.969531 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.969536 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.969544 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.969545 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.969547 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.969549 (+ 2us) write_transaction.cc:95] Timestamp: L: 3043 I1103 05:10:10.005193 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 2284.2 usec... I1103 05:10:10.008167 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 18236.1 usec... I1103 05:10:10.026621 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 42202.6 usec... I1103 05:10:10.030488 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 39477.7 usec... I1103 05:10:10.054646 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 43666 usec... I1103 05:10:10.069042 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 20658.3 usec... I1103 05:10:10.070389 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 2458.53 usec... I1103 05:10:10.090239 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 532.14 usec... I1103 05:10:10.090991 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 11260.3 usec... I1103 05:10:10.098639 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 20246.8 usec... I1103 05:10:10.098809 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 12960.9 usec... W1103 05:10:10.102520 8085 log.cc:507] Time spent Append to log took a long time: real 0.366s user 0.000s sys 0.000s W1103 05:10:10.102582 7579 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 527) took 422ms (client timeout 50). W1103 05:10:10.103304 7579 inbound_call.cc:140] Trace: 1103 05:10:09.680470 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:09.680490 (+ 20us) service_pool.cc:152] Handling call 1103 05:10:09.735862 (+ 55372us) raft_consensus.cc:579] Updating replica for 86 ops 1103 05:10:09.735867 (+ 5us) raft_consensus.cc:614] Triggering prepare for 17 ops 1103 05:10:09.736115 (+ 248us) log.cc:406] Serialized 8552 byte log entry 1103 05:10:09.736129 (+ 14us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1724 1103 05:10:09.736214 (+ 85us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1726 1103 05:10:09.736216 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:10.102561 (+366345us) raft_consensus.cc:697] finished 1103 05:10:10.102563 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:10.102565 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:10.102571 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:09.735898 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.735904 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.735905 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.735906 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.735915 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.735926 (+ 11us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.735927 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.735930 (+ 3us) write_transaction.cc:93] Start() 1103 05:10:09.735933 (+ 3us) write_transaction.cc:95] Timestamp: L: 2978 1103 05:10:09.736207 (+ 274us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736215 (+ 8us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736221 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736225 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736228 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736234 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736237 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736241 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736244 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736247 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736250 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736253 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736256 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736301 (+ 45us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.735936 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.735939 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.735940 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.735942 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.735949 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.735959 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.735960 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.735962 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.735964 (+ 2us) write_transaction.cc:95] Timestamp: L: 2979 1103 05:10:09.736216 (+ 252us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736224 (+ 8us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736227 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736230 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736232 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736235 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736239 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736241 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736243 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736246 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736249 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736251 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736255 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736260 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.735966 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.735969 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.735970 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.735971 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.735980 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.735989 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.735990 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.735992 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.735994 (+ 2us) write_transaction.cc:95] Timestamp: L: 2980 1103 05:10:09.736219 (+ 225us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736224 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736229 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736232 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736235 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736242 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736247 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736250 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736253 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736256 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736260 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736262 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736267 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736274 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.735997 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736000 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736001 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736002 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736009 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736019 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736020 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736022 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.736024 (+ 2us) write_transaction.cc:95] Timestamp: L: 2981 1103 05:10:09.736230 (+ 206us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736234 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736237 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736240 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736244 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736249 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736252 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736255 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736258 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736261 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736265 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736268 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736272 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736277 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.736026 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736029 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736030 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736031 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736037 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736045 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736047 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736048 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:09.736050 (+ 2us) write_transaction.cc:95] Timestamp: L: 2982 1103 05:10:09.736233 (+ 183us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736240 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736243 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736246 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736250 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736256 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736260 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736263 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736266 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736270 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736273 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736277 (+ 4us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736282 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736289 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.736053 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736055 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736057 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736058 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736064 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736073 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736074 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736076 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.736078 (+ 2us) write_transaction.cc:95] Timestamp: L: 2983 1103 05:10:09.736251 (+ 173us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736255 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736258 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736261 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736265 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736269 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736272 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736275 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736280 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736283 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736287 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736290 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736293 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736299 (+ 6us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.736081 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736084 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736085 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736086 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736093 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736105 (+ 12us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736106 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736108 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.736111 (+ 3us) write_transaction.cc:95] Timestamp: L: 2984 1103 05:10:09.736265 (+ 154us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736268 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736270 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736273 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736276 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736279 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736281 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736283 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736286 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736288 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736290 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736293 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736295 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736301 (+ 6us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.736113 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736116 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736117 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736118 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736125 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736134 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736136 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736137 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:09.736140 (+ 3us) write_transaction.cc:95] Timestamp: L: 2985 1103 05:10:09.736280 (+ 140us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736284 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736287 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736290 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736293 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736296 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736300 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736303 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736307 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736310 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736314 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736317 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736321 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736326 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.736142 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736145 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736147 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736148 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736166 (+ 18us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736175 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736176 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736177 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:09.736180 (+ 3us) write_transaction.cc:95] Timestamp: L: 2986 1103 05:10:09.736284 (+ 104us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736288 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736291 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736294 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736297 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736302 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736306 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736309 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736312 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736316 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736319 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736322 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736326 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736331 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.736182 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736186 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736187 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736188 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736195 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736203 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736204 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736206 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.736208 (+ 2us) write_transaction.cc:95] Timestamp: L: 2987 1103 05:10:09.736293 (+ 85us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736299 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736302 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736308 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736311 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736315 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736318 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736321 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736324 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736327 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736331 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736333 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736336 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736342 (+ 6us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:09.736215 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736218 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736220 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736221 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736228 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736239 (+ 11us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736240 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736242 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:09.736244 (+ 2us) write_transaction.cc:95] Timestamp: L: 2988 1103 05:10:09.736308 (+ 64us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736311 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736315 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736318 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736322 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736327 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736331 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736334 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736338 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736341 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736344 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736348 (+ 4us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:09.736352 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:09.736357 (+ 5us) log.cc:406] Serialized 86 byte log entry 1103 05:10:10.103003 (+366646us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:10.103005 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:10.103006 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:09.736247 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:09.736250 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:09.736252 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:09.736253 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:09.736260 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:09.736268 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:09.736270 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:09.736271 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:09.736273 (+ 2us) write_transaction.cc:95] Timestamp: L: 2989 1103 05:10:09.736317 (+ 44us) write_transaction.cc:102] APPLY: Starting 1103 05:10:09.736322 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736325 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736328 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736331 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736336 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736339 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:09.736342 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 11 I1103 05:10:10.106011 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16185.4 usec... I1103 05:10:10.131017 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 37768.3 usec... I1103 05:10:10.148697 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 32091.1 usec... I1103 05:10:10.181386 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16152.4 usec... I1103 05:10:10.197939 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 12104.7 usec... I1103 05:10:10.220784 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 8357.74 usec... I1103 05:10:10.229980 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12252.9 usec... I1103 05:10:10.238457 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 23014.9 usec... I1103 05:10:10.239238 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 39380 usec... W1103 05:10:10.246695 31061 thread.cc:320] Waited for 5000ms trying to join with ts-test0 (tid 8513) I1103 05:10:10.278847 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 37583.6 usec... W1103 05:10:10.287832 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.1994-1.1994] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:10.289872 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 7429.65 usec... W1103 05:10:10.316747 8085 log.cc:507] Time spent Append to log took a long time: real 0.080s user 0.000s sys 0.000s W1103 05:10:10.316835 7587 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 548) took 80ms (client timeout 50). W1103 05:10:10.316900 7587 inbound_call.cc:140] Trace: 1103 05:10:10.236788 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:10.236806 (+ 18us) service_pool.cc:152] Handling call 1103 05:10:10.236818 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:10.236820 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:10.236849 (+ 29us) log.cc:406] Serialized 506 byte log entry 1103 05:10:10.236856 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 1993 1103 05:10:10.236873 (+ 17us) raft_consensus.cc:682] Updating last received op as term: 1 index: 1994 1103 05:10:10.236874 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:10.316808 (+ 79934us) raft_consensus.cc:697] finished 1103 05:10:10.316811 (+ 3us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:10.316813 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:10.316821 (+ 8us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:10.236853 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:10.236858 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:10.236859 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:10.236860 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:10.236866 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:10.236873 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:10.236874 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:10.236875 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:10.236878 (+ 3us) write_transaction.cc:95] Timestamp: L: 3475 I1103 05:10:10.316896 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 1634.68 usec... I1103 05:10:10.337196 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 53367.1 usec... I1103 05:10:10.387877 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 8617.14 usec... W1103 05:10:10.390882 8084 log.cc:507] Time spent Append to log took a long time: real 0.054s user 0.000s sys 0.000s W1103 05:10:10.396875 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2123-1.2126] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:10.507227 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 36658.8 usec... I1103 05:10:10.509917 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 33790.2 usec... I1103 05:10:10.543808 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 13515.2 usec... I1103 05:10:10.544183 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 26651.2 usec... I1103 05:10:10.544352 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 34617.6 usec... W1103 05:10:10.558384 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2631-1.2635] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:10.560654 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2646-1.2646] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:10.571599 7782 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 524) took 64ms (client timeout 50). W1103 05:10:10.571732 7782 inbound_call.cc:140] Trace: 1103 05:10:10.506812 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:10.506859 (+ 47us) service_pool.cc:152] Handling call 1103 05:10:10.506881 (+ 22us) raft_consensus.cc:579] Updating replica for 5 ops 1103 05:10:10.506884 (+ 3us) raft_consensus.cc:614] Triggering prepare for 5 ops 1103 05:10:10.506934 (+ 50us) log.cc:406] Serialized 2510 byte log entry 1103 05:10:10.506940 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 2631 1103 05:10:10.506955 (+ 15us) raft_consensus.cc:682] Updating last received op as term: 1 index: 2635 1103 05:10:10.506957 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:10.571569 (+ 64612us) raft_consensus.cc:697] finished 1103 05:10:10.571571 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:10.571574 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:10:10.571580 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:10.507036 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:10.507040 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:10.507041 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:10.507042 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:10.507048 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:10.507054 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:10.507055 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:10.507056 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:10.507059 (+ 3us) write_transaction.cc:95] Timestamp: L: 4655 1103 05:10:10.507072 (+ 13us) write_transaction.cc:102] APPLY: Starting 1103 05:10:10.507077 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507080 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507084 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507087 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507090 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507093 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507096 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507098 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507101 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507106 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:10.507110 (+ 4us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:10.507115 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:10.507120 (+ 5us) log.cc:406] Serialized 86 byte log entry 1103 05:10:10.571602 (+ 64482us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:10.571604 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:10.571605 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:10.507065 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:10.507067 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:10.507068 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:10.507069 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:10.507073 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:10.507078 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:10.507079 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:10.507080 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:10.507082 (+ 2us) write_transaction.cc:95] Timestamp: L: 4656 Related trace: 1103 05:10:10.507083 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:10.507085 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:10.507086 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:10.507087 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:10.507101 (+ 14us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:10.507106 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:10.507107 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:10.507109 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:10.507110 (+ 1us) write_transaction.cc:95] Timestamp: L: 4657 Related trace: 1103 05:10:10.507112 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:10.507115 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:10.507116 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:10.507118 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:10.507124 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:10.507131 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:10.507132 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:10.507134 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:10.507137 (+ 3us) write_transaction.cc:95] Timestamp: L: 4658 Related trace: 1103 05:10:10.507139 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:10.507142 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:10.507143 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:10.507145 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:10.507151 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:10.507158 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:10.507159 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:10.507161 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:10.507163 (+ 2us) write_transaction.cc:95] Timestamp: L: 4659 I1103 05:10:10.579327 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 3382.92 usec... I1103 05:10:10.585968 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 7804.14 usec... I1103 05:10:10.592677 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 1089.72 usec... I1103 05:10:10.594521 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 31593.7 usec... I1103 05:10:10.604035 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 18206.7 usec... I1103 05:10:10.620620 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 4724.38 usec... I1103 05:10:10.629999 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 24312.3 usec... I1103 05:10:10.635182 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 40059.1 usec... I1103 05:10:10.654563 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 36033.8 usec... I1103 05:10:10.676152 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 26097.6 usec... I1103 05:10:10.691206 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15195.2 usec... I1103 05:10:10.706899 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 2488.78 usec... I1103 05:10:10.708808 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 47457.6 usec... I1103 05:10:10.747110 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 5152.27 usec... I1103 05:10:10.756440 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 27109.6 usec... I1103 05:10:10.783804 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 42796.6 usec... I1103 05:10:10.827203 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 24227.8 usec... I1103 05:10:10.849228 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 36746.2 usec... I1103 05:10:10.851650 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 31356 usec... I1103 05:10:10.854953 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 49816.9 usec... I1103 05:10:10.886370 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 18581.8 usec... I1103 05:10:10.923671 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 81552.6 usec... I1103 05:10:10.924280 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 4997.7 usec... I1103 05:10:10.927444 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 16203.9 usec... I1103 05:10:10.929500 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 54350.9 usec... I1103 05:10:10.943864 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 355.703 usec... I1103 05:10:10.953604 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 21850 usec... I1103 05:10:10.975836 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 16910.6 usec... I1103 05:10:10.992961 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 31848.2 usec... I1103 05:10:11.005450 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 13136.8 usec... I1103 05:10:11.038980 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 6566.91 usec... I1103 05:10:11.046259 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 29038.3 usec... I1103 05:10:11.052892 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 6379.09 usec... I1103 05:10:11.059505 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 44651.3 usec... I1103 05:10:11.064453 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 3941.73 usec... I1103 05:10:11.068589 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 87156.5 usec... I1103 05:10:11.075543 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 41781.8 usec... I1103 05:10:11.104656 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 9695.61 usec... I1103 05:10:11.114783 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 14880.5 usec... W1103 05:10:11.121956 8085 log.cc:507] Time spent Append to log took a long time: real 0.103s user 0.000s sys 0.000s W1103 05:10:11.122076 7579 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 809) took 102ms (client timeout 50). W1103 05:10:11.122194 7579 inbound_call.cc:140] Trace: 1103 05:10:11.019076 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:11.019153 (+ 77us) service_pool.cc:152] Handling call 1103 05:10:11.019193 (+ 40us) raft_consensus.cc:579] Updating replica for 8 ops 1103 05:10:11.019196 (+ 3us) raft_consensus.cc:614] Triggering prepare for 7 ops 1103 05:10:11.019290 (+ 94us) log.cc:406] Serialized 3522 byte log entry 1103 05:10:11.019301 (+ 11us) raft_consensus.cc:669] Marking committed up to term: 1 index: 2645 1103 05:10:11.019314 (+ 13us) raft_consensus.cc:682] Updating last received op as term: 1 index: 2653 1103 05:10:11.019315 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:11.122051 (+102736us) raft_consensus.cc:697] finished 1103 05:10:11.122053 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:11.122055 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:11.122062 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:11.019226 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.019233 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.019235 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.019236 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.019245 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.019256 (+ 11us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.019257 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.019259 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:11.019262 (+ 3us) write_transaction.cc:95] Timestamp: L: 4741 Related trace: 1103 05:10:11.019264 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.019267 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.019268 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.019269 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.019275 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.019282 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.019283 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.019284 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:11.019285 (+ 1us) write_transaction.cc:95] Timestamp: L: 4742 Related trace: 1103 05:10:11.019287 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.019290 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.019291 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.019292 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.019297 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.019304 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.019305 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.019306 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:11.019307 (+ 1us) write_transaction.cc:95] Timestamp: L: 4743 Related trace: 1103 05:10:11.019309 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.019311 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.019312 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.019313 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.019318 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.019325 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.019326 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.019327 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:11.019329 (+ 2us) write_transaction.cc:95] Timestamp: L: 4744 Related trace: 1103 05:10:11.019330 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.019332 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.019333 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.019334 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.019342 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.019350 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.019351 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.019353 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:11.019355 (+ 2us) write_transaction.cc:95] Timestamp: L: 4745 Related trace: 1103 05:10:11.019357 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.019359 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.019360 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.019361 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.019366 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.019374 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.019375 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.019376 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:11.019377 (+ 1us) write_transaction.cc:95] Timestamp: L: 4746 Related trace: 1103 05:10:11.019379 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.019381 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.019382 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.019383 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.019388 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.019395 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.019396 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.019397 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:11.019398 (+ 1us) write_transaction.cc:95] Timestamp: L: 4747 I1103 05:10:11.130244 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15036.4 usec... W1103 05:10:11.156002 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2646-1.2653] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:11.156180 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 22743.4 usec... I1103 05:10:11.168946 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 4139.66 usec... W1103 05:10:11.179216 8093 log.cc:507] Time spent Append to log took a long time: real 0.131s user 0.000s sys 0.000s I1103 05:10:11.179975 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 12255.7 usec... I1103 05:10:11.201894 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 1731.59 usec... I1103 05:10:11.203848 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 1850.13 usec... I1103 05:10:11.206745 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 23237.4 usec... I1103 05:10:11.230207 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 49702.8 usec... W1103 05:10:11.246839 31061 thread.cc:320] Waited for 6000ms trying to join with ts-test0 (tid 8513) W1103 05:10:11.257614 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2698-1.2698] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:11.280144 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 34134.1 usec... I1103 05:10:11.304615 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 7499.77 usec... W1103 05:10:11.312404 8093 log.cc:507] Time spent Append to log took a long time: real 0.050s user 0.000s sys 0.000s W1103 05:10:11.312436 7783 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 609) took 49ms (client timeout 50). W1103 05:10:11.312484 7783 inbound_call.cc:140] Trace: 1103 05:10:11.262570 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:11.262590 (+ 20us) service_pool.cc:152] Handling call 1103 05:10:11.262600 (+ 10us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:11.262602 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:11.262622 (+ 20us) log.cc:406] Serialized 496 byte log entry 1103 05:10:11.262627 (+ 5us) raft_consensus.cc:669] Marking committed up to term: 1 index: 2874 1103 05:10:11.262635 (+ 8us) raft_consensus.cc:682] Updating last received op as term: 1 index: 2875 1103 05:10:11.262637 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:11.312417 (+ 49780us) raft_consensus.cc:697] finished 1103 05:10:11.312418 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:11.312420 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:11.312425 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:11.262645 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.262649 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.262650 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.262651 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.262656 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.262663 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.262664 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.262665 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:11.262667 (+ 2us) write_transaction.cc:95] Timestamp: L: 5109 W1103 05:10:11.313143 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2875-1.2875] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:11.313448 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2698-1.2875] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:11.315515 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 15632.2 usec... I1103 05:10:11.331370 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 1949.23 usec... I1103 05:10:11.340625 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 100895 usec... I1103 05:10:11.377903 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 74148.4 usec... I1103 05:10:11.427974 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 39254.4 usec... I1103 05:10:11.441794 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 19649.2 usec... I1103 05:10:11.452272 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 24799.2 usec... I1103 05:10:11.461679 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 2474.45 usec... W1103 05:10:11.464433 8093 log.cc:507] Time spent Append to log took a long time: real 0.146s user 0.000s sys 0.000s I1103 05:10:11.464699 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 7939.44 usec... I1103 05:10:11.467469 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 17122.8 usec... I1103 05:10:11.472870 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 7397.3 usec... I1103 05:10:11.477303 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 33880.2 usec... I1103 05:10:11.511425 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 11288 usec... I1103 05:10:11.514799 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 17125.8 usec... W1103 05:10:11.532176 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2883-1.2883] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:11.537899 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 18226.6 usec... I1103 05:10:11.556351 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 38672.5 usec... W1103 05:10:11.557797 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2993-1.2993] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:11.577049 8085 log.cc:507] Time spent Append to log took a long time: real 0.069s user 0.000s sys 0.000s W1103 05:10:11.577291 7585 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 870) took 69ms (client timeout 50). W1103 05:10:11.577361 7585 inbound_call.cc:140] Trace: 1103 05:10:11.508064 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:11.508079 (+ 15us) service_pool.cc:152] Handling call 1103 05:10:11.508088 (+ 9us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:11.508090 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:11.508109 (+ 19us) log.cc:406] Serialized 506 byte log entry 1103 05:10:11.508115 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 2992 1103 05:10:11.508137 (+ 22us) raft_consensus.cc:682] Updating last received op as term: 1 index: 2993 1103 05:10:11.508138 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:11.577267 (+ 69129us) raft_consensus.cc:697] finished 1103 05:10:11.577269 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:11.577271 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:11.577278 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:11.508147 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:11.508150 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:11.508151 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:11.508152 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:11.508158 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:11.508165 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:11.508166 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:11.508167 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:11.508170 (+ 3us) write_transaction.cc:95] Timestamp: L: 5247 I1103 05:10:11.585906 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 58084 usec... W1103 05:10:11.596258 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2883-1.2997] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:11.644223 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 51602.3 usec... I1103 05:10:11.651321 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 16673.1 usec... I1103 05:10:11.695446 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 29821.3 usec... I1103 05:10:11.696063 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 24707.2 usec... I1103 05:10:11.722450 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 6848.07 usec... I1103 05:10:11.729832 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 15724.7 usec... I1103 05:10:11.751937 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 18539.2 usec... I1103 05:10:11.760622 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 41357.8 usec... I1103 05:10:11.771196 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 19714.9 usec... I1103 05:10:11.792001 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 27063.7 usec... I1103 05:10:11.799479 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 20414.4 usec... I1103 05:10:11.819293 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 35462.9 usec... I1103 05:10:11.820114 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16260.3 usec... I1103 05:10:11.849198 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 51422.4 usec... I1103 05:10:11.886237 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 39197 usec... I1103 05:10:11.900861 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 13.7024 usec... I1103 05:10:11.912181 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 42355.9 usec... I1103 05:10:11.925668 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 32676.3 usec... I1103 05:10:11.925736 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 20317.9 usec... I1103 05:10:11.958811 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 52395.8 usec... I1103 05:10:11.985072 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 16404.4 usec... I1103 05:10:12.002717 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 10894 usec... I1103 05:10:12.007552 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 4598.07 usec... I1103 05:10:12.011432 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 52266.5 usec... I1103 05:10:12.012789 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 8091.23 usec... I1103 05:10:12.013938 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 15723.2 usec... I1103 05:10:12.020998 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 6043.87 usec... I1103 05:10:12.033799 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 5361.34 usec... I1103 05:10:12.064419 7546 maintenance_manager.cc:317] Time spent running LogGCOp: real 0.000s user 0.000s sys 0.000s I1103 05:10:12.064519 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 36862.2 usec... I1103 05:10:12.077203 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 10703.2 usec... W1103 05:10:12.080823 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.2993-1.3000] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:12.084069 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 34252.2 usec... I1103 05:10:12.088127 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 15517.2 usec... I1103 05:10:12.102339 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 18900.6 usec... I1103 05:10:12.118010 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 24147.9 usec... I1103 05:10:12.119482 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 24392.3 usec... I1103 05:10:12.121451 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 35388.3 usec... I1103 05:10:12.142804 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 33303.4 usec... W1103 05:10:12.144647 8093 log.cc:507] Time spent Append to log took a long time: real 0.067s user 0.000s sys 0.000s I1103 05:10:12.157110 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 12183.7 usec... W1103 05:10:12.169531 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3041-1.3046] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:12.207767 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 13928.9 usec... I1103 05:10:12.208652 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 32155.6 usec... I1103 05:10:12.214879 7596 maintenance_manager.cc:317] Time spent running LogGCOp: real 0.000s user 0.000s sys 0.000s I1103 05:10:12.221927 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 3638.21 usec... I1103 05:10:12.226107 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 19845.7 usec... I1103 05:10:12.241292 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 5388.31 usec... I1103 05:10:12.246170 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 9313.73 usec... I1103 05:10:12.247025 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 5010.56 usec... W1103 05:10:12.247112 31061 thread.cc:320] Waited for 7000ms trying to join with ts-test0 (tid 8513) I1103 05:10:12.247124 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 24454.7 usec... W1103 05:10:12.258502 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3065-1.3065] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:12.272017 7786 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 634) took 63ms (client timeout 50). W1103 05:10:12.272095 7786 inbound_call.cc:140] Trace: 1103 05:10:12.208493 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:12.208519 (+ 26us) service_pool.cc:152] Handling call 1103 05:10:12.208534 (+ 15us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:12.208537 (+ 3us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:12.208572 (+ 35us) log.cc:406] Serialized 496 byte log entry 1103 05:10:12.208580 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3064 1103 05:10:12.208625 (+ 45us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3065 1103 05:10:12.208627 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:12.271993 (+ 63366us) raft_consensus.cc:697] finished 1103 05:10:12.271995 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:12.271997 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:12.272003 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:12.208572 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.208578 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.208579 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.208581 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.208588 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.208597 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.208599 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.208600 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.208604 (+ 4us) write_transaction.cc:95] Timestamp: L: 5438 I1103 05:10:12.291409 7751 maintenance_manager.cc:317] Time spent running LogGCOp: real 0.000s user 0.000s sys 0.000s I1103 05:10:12.302451 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 23239.6 usec... I1103 05:10:12.321455 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 1268.2 usec... I1103 05:10:12.324404 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 38964.2 usec... W1103 05:10:12.326915 7787 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 640) took 46ms (client timeout 50). W1103 05:10:12.326972 7787 inbound_call.cc:140] Trace: 1103 05:10:12.280053 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:12.280080 (+ 27us) service_pool.cc:152] Handling call 1103 05:10:12.280096 (+ 16us) raft_consensus.cc:579] Updating replica for 2 ops 1103 05:10:12.280098 (+ 2us) raft_consensus.cc:614] Triggering prepare for 2 ops 1103 05:10:12.280139 (+ 41us) log.cc:406] Serialized 1002 byte log entry 1103 05:10:12.280147 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3171 1103 05:10:12.280178 (+ 31us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3173 1103 05:10:12.280180 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:12.326893 (+ 46713us) raft_consensus.cc:697] finished 1103 05:10:12.326894 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:12.326896 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:12.326902 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:12.280130 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.280135 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.280136 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.280138 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.280145 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.280154 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.280155 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.280157 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.280160 (+ 3us) write_transaction.cc:95] Timestamp: L: 5625 Related trace: 1103 05:10:12.280162 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.280166 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.280167 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.280168 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.280174 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.280181 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.280183 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.280184 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.280187 (+ 3us) write_transaction.cc:95] Timestamp: L: 5626 I1103 05:10:12.332592 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 6380.67 usec... I1103 05:10:12.333117 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 32883.9 usec... W1103 05:10:12.364068 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3172-1.3173] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:12.364411 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 19203.5 usec... I1103 05:10:12.366224 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 11613.4 usec... I1103 05:10:12.398342 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 73278.6 usec... I1103 05:10:12.398684 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 17811.9 usec... I1103 05:10:12.414690 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 23384.6 usec... I1103 05:10:12.417043 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 22561.8 usec... I1103 05:10:12.438261 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 8992.27 usec... I1103 05:10:12.439852 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 683.234 usec... I1103 05:10:12.448181 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 17998.1 usec... I1103 05:10:12.466423 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 22645.4 usec... W1103 05:10:12.471848 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3243-1.3244] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:12.472091 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3172-1.3188] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:12.479028 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 3473.62 usec... I1103 05:10:12.494431 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 18740.1 usec... I1103 05:10:12.499295 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12343.3 usec... I1103 05:10:12.512321 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 19451.4 usec... I1103 05:10:12.531991 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12145.1 usec... I1103 05:10:12.545317 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 20858 usec... I1103 05:10:12.553356 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 8318.68 usec... I1103 05:10:12.561972 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 40871.4 usec... W1103 05:10:12.566440 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3272-1.3277] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:12.566478 8093 log.cc:507] Time spent Append to log took a long time: real 0.053s user 0.000s sys 0.000s W1103 05:10:12.566443 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3271-1.3276] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:12.635506 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 27600.9 usec... I1103 05:10:12.639611 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 2743.64 usec... I1103 05:10:12.641764 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 5482.6 usec... I1103 05:10:12.642848 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 55338.9 usec... I1103 05:10:12.647471 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16499.4 usec... I1103 05:10:12.664202 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 17581.3 usec... I1103 05:10:12.682003 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 75824.2 usec... I1103 05:10:12.698405 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 4267.6 usec... W1103 05:10:12.702905 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3339-1.3343] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:12.702922 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3305-1.3342] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:12.703060 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 30825.3 usec... I1103 05:10:12.727149 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 43298.7 usec... W1103 05:10:12.734163 8093 log.cc:507] Time spent Append to log took a long time: real 0.121s user 0.000s sys 0.000s W1103 05:10:12.758191 8085 log.cc:507] Time spent Append to log took a long time: real 0.144s user 0.000s sys 0.000s W1103 05:10:12.758877 7584 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 962) took 145ms (client timeout 50). W1103 05:10:12.760258 7584 inbound_call.cc:140] Trace: 1103 05:10:12.613321 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:12.613345 (+ 24us) service_pool.cc:152] Handling call 1103 05:10:12.613465 (+ 120us) raft_consensus.cc:579] Updating replica for 38 ops 1103 05:10:12.613468 (+ 3us) raft_consensus.cc:614] Triggering prepare for 38 ops 1103 05:10:12.613883 (+ 415us) log.cc:406] Serialized 19138 byte log entry 1103 05:10:12.613891 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3336 1103 05:10:12.613971 (+ 80us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3342 1103 05:10:12.613973 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:12.758854 (+144881us) raft_consensus.cc:697] finished 1103 05:10:12.758855 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:12.758857 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:12.758866 (+ 9us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:12.613499 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613504 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613506 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613507 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613516 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613526 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613527 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613529 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613532 (+ 3us) write_transaction.cc:95] Timestamp: L: 5803 1103 05:10:12.614126 (+ 594us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614131 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614133 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614136 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614139 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614153 (+ 14us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614156 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614159 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614163 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614167 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614170 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614172 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614177 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614182 (+ 5us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758909 (+144727us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758910 (+ 1us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758912 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613538 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613541 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613543 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613544 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613552 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613561 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613562 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613564 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613567 (+ 3us) write_transaction.cc:95] Timestamp: L: 5808 1103 05:10:12.614159 (+ 592us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614163 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614167 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614171 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614173 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614177 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614182 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614185 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614188 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614190 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614193 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614195 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614199 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614203 (+ 4us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758925 (+144722us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758927 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758928 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613569 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613573 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613574 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613575 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613583 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613592 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613593 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613595 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613598 (+ 3us) write_transaction.cc:95] Timestamp: L: 5810 1103 05:10:12.614185 (+ 587us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614189 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614191 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614194 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614199 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614203 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614206 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614216 (+ 10us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614219 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614222 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614225 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614227 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614231 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614238 (+ 7us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758933 (+144695us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758935 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758936 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613600 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613604 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613605 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613606 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613613 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613621 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613622 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613624 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613627 (+ 3us) write_transaction.cc:95] Timestamp: L: 5811 1103 05:10:12.614193 (+ 566us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614200 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614203 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614214 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614218 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614223 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614227 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614230 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614233 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614236 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614240 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614243 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614247 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614254 (+ 7us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758949 (+144695us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758950 (+ 1us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758951 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613629 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613632 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613634 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613635 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613641 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613649 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613650 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613652 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613655 (+ 3us) write_transaction.cc:95] Timestamp: L: 5812 1103 05:10:12.614193 (+ 538us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614198 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614202 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614207 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614218 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614222 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614225 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614229 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614232 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614235 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614240 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614243 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614246 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614252 (+ 6us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758942 (+144690us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758943 (+ 1us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758944 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613658 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613661 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613663 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613664 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613670 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613678 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613679 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613681 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613683 (+ 2us) write_transaction.cc:95] Timestamp: L: 5813 1103 05:10:12.614207 (+ 524us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614216 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614219 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614223 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614228 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614233 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614235 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614239 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614242 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614245 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614248 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614250 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614253 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614257 (+ 4us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758956 (+144699us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758958 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758959 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613686 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613689 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613690 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613691 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613697 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613705 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613707 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613708 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.613711 (+ 3us) write_transaction.cc:95] Timestamp: L: 5814 1103 05:10:12.614242 (+ 531us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614245 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614248 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614250 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614253 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614257 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614260 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614263 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614266 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614275 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614278 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614280 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614284 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614288 (+ 4us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758964 (+144676us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758965 (+ 1us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758966 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613713 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613716 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613718 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613719 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613724 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613732 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613733 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613735 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613738 (+ 3us) write_transaction.cc:95] Timestamp: L: 5815 1103 05:10:12.614256 (+ 518us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614261 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614264 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614267 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614276 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614294 (+ 18us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614296 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614298 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614302 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614304 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614306 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614308 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614312 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614317 (+ 5us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758978 (+144661us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758980 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758981 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613740 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613744 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613745 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613746 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613755 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613762 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613764 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613765 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.613768 (+ 3us) write_transaction.cc:95] Timestamp: L: 5816 1103 05:10:12.614259 (+ 491us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614266 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614269 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614280 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614283 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614288 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614291 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614295 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614298 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614301 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614304 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614307 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614311 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614320 (+ 9us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758986 (+144666us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758987 (+ 1us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758988 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613770 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613774 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613775 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613776 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613782 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613790 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613792 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613794 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613796 (+ 2us) write_transaction.cc:95] Timestamp: L: 5817 1103 05:10:12.614262 (+ 466us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614265 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614268 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614277 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614280 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614287 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614290 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614293 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614296 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614298 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614301 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614303 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.614310 (+ 7us) write_transaction.cc:228] Released schema lock 1103 05:10:12.614314 (+ 4us) log.cc:406] Serialized 86 byte log entry 1103 05:10:12.758970 (+144656us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:12.758972 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:12.758973 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:12.613800 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.613804 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.613805 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.613807 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.613813 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.613822 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.613823 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.613825 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.613827 (+ 2us) write_transaction.cc:95] Timestamp: L: 5818 1103 05:10:12.614291 (+ 464us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.614295 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614298 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614301 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614305 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614309 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614311 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614314 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614317 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.614320 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_co I1103 05:10:12.770684 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 5455.7 usec... W1103 05:10:12.776417 8084 log.cc:507] Time spent Append to log took a long time: real 0.162s user 0.000s sys 0.000s W1103 05:10:12.776540 7784 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 670) took 162ms (client timeout 50). W1103 05:10:12.776625 7784 inbound_call.cc:140] Trace: 1103 05:10:12.613834 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:12.613861 (+ 27us) service_pool.cc:152] Handling call 1103 05:10:12.613886 (+ 25us) raft_consensus.cc:579] Updating replica for 5 ops 1103 05:10:12.613889 (+ 3us) raft_consensus.cc:614] Triggering prepare for 5 ops 1103 05:10:12.613961 (+ 72us) log.cc:406] Serialized 2510 byte log entry 1103 05:10:12.613967 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3338 1103 05:10:12.613985 (+ 18us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3343 1103 05:10:12.613987 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:12.776509 (+162522us) raft_consensus.cc:697] finished 1103 05:10:12.776511 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:12.776513 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:12.776525 (+ 12us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:12.614283 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.614288 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.614290 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.614291 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.614300 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.614308 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.614310 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.614312 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.614315 (+ 3us) write_transaction.cc:95] Timestamp: L: 5929 Related trace: 1103 05:10:12.614318 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.614321 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.614322 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.614323 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.614329 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.614337 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.614338 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.614340 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.614342 (+ 2us) write_transaction.cc:95] Timestamp: L: 5930 Related trace: 1103 05:10:12.614345 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.614348 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.614349 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.614351 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.614357 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.614363 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.614365 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.614366 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.614368 (+ 2us) write_transaction.cc:95] Timestamp: L: 5931 Related trace: 1103 05:10:12.614370 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.614374 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.614375 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.614376 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.614382 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.614390 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.614391 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.614393 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.614395 (+ 2us) write_transaction.cc:95] Timestamp: L: 5932 Related trace: 1103 05:10:12.614397 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.614400 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.614402 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.614403 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.614409 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.614416 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.614417 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.614419 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.614422 (+ 3us) write_transaction.cc:95] Timestamp: L: 5933 I1103 05:10:12.778259 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 52126 usec... I1103 05:10:12.786522 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 28015.3 usec... I1103 05:10:12.825063 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 39658.9 usec... I1103 05:10:12.830615 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 16665.1 usec... W1103 05:10:12.847612 7583 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 965) took 69ms (client timeout 50). W1103 05:10:12.847681 7583 inbound_call.cc:140] Trace: 1103 05:10:12.778248 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:12.778268 (+ 20us) service_pool.cc:152] Handling call 1103 05:10:12.847495 (+ 69227us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:12.847498 (+ 3us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:12.847537 (+ 39us) log.cc:406] Serialized 506 byte log entry 1103 05:10:12.847544 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3346 1103 05:10:12.847556 (+ 12us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3347 1103 05:10:12.847557 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:12.847592 (+ 35us) raft_consensus.cc:697] finished 1103 05:10:12.847593 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:12.847595 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:12.847601 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:12.847543 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.847550 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.847552 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.847552 (+ 0us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.847561 (+ 9us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.847587 (+ 26us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.847588 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.847590 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.847593 (+ 3us) write_transaction.cc:95] Timestamp: L: 5884 W1103 05:10:12.864873 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3347-1.3347] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:12.890710 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 54458.7 usec... I1103 05:10:12.908089 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 5534.52 usec... I1103 05:10:12.913839 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 31552.4 usec... W1103 05:10:12.927207 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3432-1.3434] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:12.927500 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3396-1.3432] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:12.933527 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 93784 usec... W1103 05:10:12.945655 8085 log.cc:507] Time spent Append to log took a long time: real 0.068s user 0.000s sys 0.000s W1103 05:10:12.945763 7579 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 981) took 68ms (client timeout 50). W1103 05:10:12.945796 8084 log.cc:507] Time spent Append to log took a long time: real 0.068s user 0.000s sys 0.001s W1103 05:10:12.945832 7579 inbound_call.cc:140] Trace: 1103 05:10:12.877025 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:12.877046 (+ 21us) service_pool.cc:152] Handling call 1103 05:10:12.877063 (+ 17us) raft_consensus.cc:579] Updating replica for 3 ops 1103 05:10:12.877065 (+ 2us) raft_consensus.cc:614] Triggering prepare for 3 ops 1103 05:10:12.877122 (+ 57us) log.cc:406] Serialized 1518 byte log entry 1103 05:10:12.877130 (+ 8us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3431 1103 05:10:12.877176 (+ 46us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3434 1103 05:10:12.877178 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:12.945741 (+ 68563us) raft_consensus.cc:697] finished 1103 05:10:12.945743 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:12.945745 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:12.945751 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:12.877349 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877367 (+ 18us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877368 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877369 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877376 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877384 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877386 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877387 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.877390 (+ 3us) write_transaction.cc:95] Timestamp: L: 6012 Related trace: 1103 05:10:12.877393 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877396 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877397 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877398 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877404 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877411 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877412 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877414 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877416 (+ 2us) write_transaction.cc:95] Timestamp: L: 6013 Related trace: 1103 05:10:12.877418 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877421 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877423 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877424 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877429 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877438 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877439 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877441 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877443 (+ 2us) write_transaction.cc:95] Timestamp: L: 6014 W1103 05:10:12.945863 7780 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 689) took 68ms (client timeout 50). I1103 05:10:12.946385 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 24208.1 usec... W1103 05:10:12.971911 7780 inbound_call.cc:140] Trace: 1103 05:10:12.876887 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:12.876912 (+ 25us) service_pool.cc:152] Handling call 1103 05:10:12.877029 (+ 117us) raft_consensus.cc:579] Updating replica for 37 ops 1103 05:10:12.877033 (+ 4us) raft_consensus.cc:614] Triggering prepare for 37 ops 1103 05:10:12.877569 (+ 536us) log.cc:406] Serialized 18622 byte log entry 1103 05:10:12.877583 (+ 14us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3427 1103 05:10:12.877668 (+ 85us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3432 1103 05:10:12.877670 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:12.945843 (+ 68173us) raft_consensus.cc:697] finished 1103 05:10:12.945844 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:12.945846 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:12.945851 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:12.877071 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877077 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877078 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877079 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877087 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877097 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877098 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877100 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877103 (+ 3us) write_transaction.cc:95] Timestamp: L: 6040 1103 05:10:12.877643 (+ 540us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877647 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877650 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877654 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877665 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877669 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877672 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877675 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877677 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877680 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877683 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877686 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877689 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877696 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877105 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877108 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877110 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877111 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877117 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877125 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877126 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877127 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.877129 (+ 2us) write_transaction.cc:95] Timestamp: L: 6041 1103 05:10:12.877646 (+ 517us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877653 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877656 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877669 (+ 13us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877673 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877678 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877681 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877685 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877688 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877691 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877694 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877697 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877701 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877708 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877131 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877134 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877135 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877136 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877143 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877231 (+ 88us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877232 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877235 (+ 3us) write_transaction.cc:93] Start() 1103 05:10:12.877238 (+ 3us) write_transaction.cc:95] Timestamp: L: 6042 1103 05:10:12.877683 (+ 445us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877689 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877693 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877697 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877700 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877706 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877709 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877713 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877716 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877720 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877736 (+ 16us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877740 (+ 4us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877744 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877751 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877241 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877245 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877247 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877248 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877256 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877265 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877266 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877268 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877270 (+ 2us) write_transaction.cc:95] Timestamp: L: 6043 1103 05:10:12.877705 (+ 435us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877710 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877713 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877717 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877720 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877735 (+ 15us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877739 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877744 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877747 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877750 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877753 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877755 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877760 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877764 (+ 4us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877273 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877276 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877277 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877279 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877285 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877293 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877294 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877296 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877298 (+ 2us) write_transaction.cc:95] Timestamp: L: 6044 1103 05:10:12.877706 (+ 408us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877710 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877713 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877716 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877719 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877723 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877733 (+ 10us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877736 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877739 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877743 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877745 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877748 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877751 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877756 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877300 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877303 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877304 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877306 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877312 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877321 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877323 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877324 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.877326 (+ 2us) write_transaction.cc:95] Timestamp: L: 6045 1103 05:10:12.877713 (+ 387us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877718 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877721 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877735 (+ 14us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877740 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877744 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877747 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877751 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877754 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877757 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877764 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877768 (+ 4us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877772 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877778 (+ 6us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877329 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877332 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877333 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877334 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877342 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877356 (+ 14us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877357 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877359 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877361 (+ 2us) write_transaction.cc:95] Timestamp: L: 6046 1103 05:10:12.877713 (+ 352us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877717 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877721 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877732 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877735 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877740 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877743 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877747 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877752 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877755 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877764 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877768 (+ 4us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877773 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877778 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877365 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877369 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877370 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877371 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877378 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877387 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877389 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877392 (+ 3us) write_transaction.cc:93] Start() 1103 05:10:12.877395 (+ 3us) write_transaction.cc:95] Timestamp: L: 6047 1103 05:10:12.877726 (+ 331us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877732 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877737 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877741 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877744 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877751 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877755 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877758 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877771 (+ 13us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877776 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877779 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877783 (+ 4us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877787 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877793 (+ 6us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877397 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877400 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877401 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877402 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877409 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877419 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877420 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877421 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:12.877423 (+ 2us) write_transaction.cc:95] Timestamp: L: 6048 1103 05:10:12.877753 (+ 330us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877759 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877774 (+ 15us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877777 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877780 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877784 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877789 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877793 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877797 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877807 (+ 10us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877811 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877813 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877819 (+ 6us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877826 (+ 7us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877426 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877429 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877430 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877431 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877438 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877447 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877449 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877451 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877453 (+ 2us) write_transaction.cc:95] Timestamp: L: 6049 1103 05:10:12.877755 (+ 302us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877766 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877772 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877775 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877778 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877784 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877788 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877792 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877795 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877806 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877809 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877812 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877817 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877823 (+ 6us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877456 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877459 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877460 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877462 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877469 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877483 (+ 14us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877484 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877486 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877488 (+ 2us) write_transaction.cc:95] Timestamp: L: 6050 1103 05:10:12.877760 (+ 272us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877764 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877773 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877776 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877779 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877784 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877787 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877791 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877794 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877804 (+ 10us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877808 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877811 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:12.877814 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:12.877819 (+ 5us) log.cc:406] Serialized 86 byte log entry Related trace: 1103 05:10:12.877490 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:12.877494 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:12.877495 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:12.877496 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:12.877503 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:12.877512 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:12.877513 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:12.877515 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:12.877517 (+ 2us) write_transaction.cc:95] Timestamp: L: 6051 1103 05:10:12.877769 (+ 252us) write_transaction.cc:102] APPLY: Starting 1103 05:10:12.877775 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877791 (+ 16us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877795 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877804 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877809 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877812 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877815 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877819 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:12.877822 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas I1103 05:10:13.024410 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 11681.2 usec... W1103 05:10:13.027823 8093 log.cc:507] Time spent Append to log took a long time: real 0.150s user 0.000s sys 0.000s I1103 05:10:13.050750 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 77186.5 usec... I1103 05:10:13.063938 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 20439.8 usec... I1103 05:10:13.084594 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 46972.6 usec... I1103 05:10:13.100739 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 8259.03 usec... W1103 05:10:13.109210 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3526-1.3526] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:13.109230 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 20226.8 usec... I1103 05:10:13.128515 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 15230.6 usec... W1103 05:10:13.129704 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3541-1.3541] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:13.129730 8093 log.cc:507] Time spent Append to log took a long time: real 0.075s user 0.000s sys 0.000s W1103 05:10:13.131821 8084 log.cc:507] Time spent Append to log took a long time: real 0.077s user 0.000s sys 0.000s W1103 05:10:13.131903 7784 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 723) took 76ms (client timeout 50). W1103 05:10:13.131952 7784 inbound_call.cc:140] Trace: 1103 05:10:13.055000 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:13.055019 (+ 19us) service_pool.cc:152] Handling call 1103 05:10:13.055031 (+ 12us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:13.055033 (+ 2us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:13.055054 (+ 21us) log.cc:406] Serialized 506 byte log entry 1103 05:10:13.055061 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3540 1103 05:10:13.055122 (+ 61us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3541 1103 05:10:13.055123 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:13.131884 (+ 76761us) raft_consensus.cc:697] finished 1103 05:10:13.131885 (+ 1us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:13.131887 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:13.131893 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:13.055059 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.055064 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.055065 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.055066 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.055094 (+ 28us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.055191 (+ 97us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.055192 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.055194 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.055197 (+ 3us) write_transaction.cc:95] Timestamp: L: 6312 I1103 05:10:13.144124 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 11226.8 usec... I1103 05:10:13.155572 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 46166.9 usec... W1103 05:10:13.180677 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3526-1.3543] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:13.183761 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 16511.7 usec... I1103 05:10:13.231796 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 5408.9 usec... I1103 05:10:13.239418 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 53025.1 usec... W1103 05:10:13.247264 31061 thread.cc:320] Waited for 8000ms trying to join with ts-test0 (tid 8513) I1103 05:10:13.249300 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 31599.5 usec... I1103 05:10:13.277497 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 82.5617 usec... W1103 05:10:13.281489 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3574-1.3580] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:13.293058 8084 log.cc:507] Time spent Append to log took a long time: real 0.092s user 0.000s sys 0.000s W1103 05:10:13.293848 7787 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 735) took 92ms (client timeout 50). W1103 05:10:13.294039 7787 inbound_call.cc:140] Trace: 1103 05:10:13.201469 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:13.201492 (+ 23us) service_pool.cc:152] Handling call 1103 05:10:13.201523 (+ 31us) raft_consensus.cc:579] Updating replica for 7 ops 1103 05:10:13.201525 (+ 2us) raft_consensus.cc:614] Triggering prepare for 7 ops 1103 05:10:13.201592 (+ 67us) log.cc:406] Serialized 3522 byte log entry 1103 05:10:13.201598 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3573 1103 05:10:13.201607 (+ 9us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3580 1103 05:10:13.201608 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:13.293822 (+ 92214us) raft_consensus.cc:697] finished 1103 05:10:13.293824 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:13.293827 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:10:13.293833 (+ 6us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:13.201549 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.201556 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.201557 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.201558 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.201564 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.201572 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.201573 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.201575 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.201577 (+ 2us) write_transaction.cc:95] Timestamp: L: 6374 Related trace: 1103 05:10:13.201580 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.201582 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.201583 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.201584 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.201588 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.201594 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.201595 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.201596 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.201598 (+ 2us) write_transaction.cc:95] Timestamp: L: 6375 Related trace: 1103 05:10:13.201599 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.201602 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.201603 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.201604 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.201608 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.201614 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.201615 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.201616 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.201617 (+ 1us) write_transaction.cc:95] Timestamp: L: 6376 Related trace: 1103 05:10:13.201619 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.201621 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.201622 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.201623 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.201627 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.201633 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.201634 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.201635 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.201636 (+ 1us) write_transaction.cc:95] Timestamp: L: 6377 Related trace: 1103 05:10:13.201640 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.201644 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.201645 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.201646 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.201650 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.201657 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.201658 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.201659 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.201661 (+ 2us) write_transaction.cc:95] Timestamp: L: 6378 Related trace: 1103 05:10:13.201663 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.201665 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.201666 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.201667 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.201671 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.201677 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.201678 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.201679 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.201680 (+ 1us) write_transaction.cc:95] Timestamp: L: 6379 Related trace: 1103 05:10:13.201682 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.201684 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.201685 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.201686 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.201690 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.201695 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.201696 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.201697 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.201698 (+ 1us) write_transaction.cc:95] Timestamp: L: 6380 I1103 05:10:13.306160 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 19050.3 usec... I1103 05:10:13.310189 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 59571.4 usec... I1103 05:10:13.328663 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 8179.68 usec... I1103 05:10:13.367921 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 76034 usec... I1103 05:10:13.370308 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 27725.2 usec... W1103 05:10:13.376993 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3616-1.3616] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:13.385206 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 23695 usec... W1103 05:10:13.418339 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3655-1.3657] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:13.430850 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 4825 usec... W1103 05:10:13.444602 8085 log.cc:507] Time spent Append to log took a long time: real 0.077s user 0.000s sys 0.000s I1103 05:10:13.544723 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 9151.37 usec... I1103 05:10:13.554119 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 63695.2 usec... I1103 05:10:13.570797 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 3609.72 usec... I1103 05:10:13.582603 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 39630.3 usec... I1103 05:10:13.582818 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 29303.7 usec... I1103 05:10:13.612570 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 12477.5 usec... I1103 05:10:13.675098 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 6082.52 usec... I1103 05:10:13.678375 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 16846.2 usec... I1103 05:10:13.695616 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 39568.8 usec... I1103 05:10:13.715548 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 26869.9 usec... I1103 05:10:13.736210 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 4759.83 usec... I1103 05:10:13.762547 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 23640 usec... I1103 05:10:13.765358 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 24946.4 usec... I1103 05:10:13.802140 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 3387.29 usec... I1103 05:10:13.805909 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 39597.2 usec... I1103 05:10:13.816113 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 32585.9 usec... I1103 05:10:13.830932 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 16909.1 usec... I1103 05:10:13.846129 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 46241.9 usec... I1103 05:10:13.849011 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 9822.58 usec... I1103 05:10:13.859566 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 56498.7 usec... W1103 05:10:13.877506 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3616-1.3662] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:13.891773 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 8163.36 usec... I1103 05:10:13.892602 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 48474.9 usec... I1103 05:10:13.916535 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 56367.1 usec... W1103 05:10:13.919390 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3655-1.3662] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:13.941318 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 57965.7 usec... I1103 05:10:13.963623 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 34341 usec... I1103 05:10:13.973166 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 405.204 usec... W1103 05:10:13.975074 7780 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 739) took 126ms (client timeout 50). W1103 05:10:13.976244 7780 inbound_call.cc:140] Trace: 1103 05:10:13.849007 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:13.849021 (+ 14us) service_pool.cc:152] Handling call 1103 05:10:13.974003 (+124982us) raft_consensus.cc:579] Updating replica for 47 ops 1103 05:10:13.974007 (+ 4us) raft_consensus.cc:614] Triggering prepare for 46 ops 1103 05:10:13.974491 (+ 484us) log.cc:406] Serialized 23156 byte log entry 1103 05:10:13.974505 (+ 14us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3654 1103 05:10:13.975045 (+ 540us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3662 1103 05:10:13.975051 (+ 6us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:13.975052 (+ 1us) raft_consensus.cc:697] finished 1103 05:10:13.975054 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:13.975055 (+ 1us) raft_consensus.cc:443] Updating watermarks 1103 05:10:13.975060 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:13.974089 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974107 (+ 18us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974108 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974110 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974131 (+ 21us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974143 (+ 12us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974144 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974146 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.974149 (+ 3us) write_transaction.cc:95] Timestamp: L: 6451 1103 05:10:13.974632 (+ 483us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.974638 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974642 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974644 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974647 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974650 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974653 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974655 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974657 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974662 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974665 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974667 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.974671 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:13.974677 (+ 6us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.974739 (+ 62us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.974740 (+ 1us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975070 (+ 330us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974152 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974155 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974156 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974156 (+ 0us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974162 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974170 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974171 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974173 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.974174 (+ 1us) write_transaction.cc:95] Timestamp: L: 6452 1103 05:10:13.974705 (+ 531us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.974712 (+ 7us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974715 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974719 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974733 (+ 14us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974739 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974742 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974745 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974749 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974754 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974758 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974761 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.974766 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:13.974791 (+ 25us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.974828 (+ 37us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.974830 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974176 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974179 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974180 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974181 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974186 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974193 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974194 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974196 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.974197 (+ 1us) write_transaction.cc:95] Timestamp: L: 6453 1103 05:10:13.974792 (+ 595us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.974797 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974799 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974824 (+ 25us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974833 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974837 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974839 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974841 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974843 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974845 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974847 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974850 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.974853 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:13.974863 (+ 10us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.974897 (+ 34us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.974899 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974199 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974201 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974202 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974203 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974208 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974215 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974216 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974217 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.974219 (+ 2us) write_transaction.cc:95] Timestamp: L: 6454 1103 05:10:13.974865 (+ 646us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.974870 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974872 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974886 (+ 14us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974888 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974892 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974894 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974896 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974898 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974902 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974907 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974909 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.974912 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:13.974923 (+ 11us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.974952 (+ 29us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.974954 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975094 (+ 140us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974221 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974224 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974225 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974226 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974230 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974236 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974237 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974239 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.974240 (+ 1us) write_transaction.cc:95] Timestamp: L: 6455 1103 05:10:13.974937 (+ 697us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.974942 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974944 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974947 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974949 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974960 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974963 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974966 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974968 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974972 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974974 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.974977 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.974982 (+ 5us) write_transaction.cc:228] Released schema lock 1103 05:10:13.974992 (+ 10us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.975021 (+ 29us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.975023 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974242 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974244 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974245 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974246 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974250 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974258 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974259 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974260 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.974262 (+ 2us) write_transaction.cc:95] Timestamp: L: 6456 1103 05:10:13.975006 (+ 744us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.975011 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975014 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975023 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975025 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975028 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975030 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975033 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975035 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975036 (+ 1us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975039 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975041 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.975045 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975054 (+ 9us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.975082 (+ 28us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.975084 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974264 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974266 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974267 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974268 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974274 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974281 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974282 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974283 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.974285 (+ 2us) write_transaction.cc:95] Timestamp: L: 6457 1103 05:10:13.975082 (+ 797us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.975086 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975095 (+ 9us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975099 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975102 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975107 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975110 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975114 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975118 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975121 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975124 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975126 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.975130 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975135 (+ 5us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.975187 (+ 52us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.975189 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975190 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974286 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974288 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974290 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974290 (+ 0us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974295 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974303 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974304 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974305 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.974307 (+ 2us) write_transaction.cc:95] Timestamp: L: 6458 1103 05:10:13.975083 (+ 776us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.975094 (+ 11us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975096 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975099 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975102 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975105 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975107 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975109 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975112 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975115 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975117 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975119 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.975121 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975130 (+ 9us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.975176 (+ 46us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.975178 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975179 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974309 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974311 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974312 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974313 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974317 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974323 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974324 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974326 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.974327 (+ 1us) write_transaction.cc:95] Timestamp: L: 6459 1103 05:10:13.975108 (+ 781us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.975114 (+ 6us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975118 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975121 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975124 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975129 (+ 5us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975132 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975135 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975138 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975142 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975146 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975149 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.975153 (+ 4us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975160 (+ 7us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.975254 (+ 94us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.975255 (+ 1us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975257 (+ 2us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974329 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974331 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974332 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974333 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974339 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974345 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974346 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974347 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.974349 (+ 2us) write_transaction.cc:95] Timestamp: L: 6460 1103 05:10:13.975134 (+ 785us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.975137 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975139 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975142 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975144 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975147 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975166 (+ 19us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975169 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975171 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975174 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975178 (+ 4us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975181 (+ 3us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.975184 (+ 3us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975190 (+ 6us) log.cc:406] Serialized 86 byte log entry 1103 05:10:13.975272 (+ 82us) write_transaction.cc:149] FINISH: making edits visible 1103 05:10:13.975274 (+ 2us) write_transaction.cc:228] Released schema lock 1103 05:10:13.975275 (+ 1us) write_transaction.cc:228] Released schema lock Related trace: 1103 05:10:13.974351 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.974353 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.974354 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.974355 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.974361 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.974369 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.974370 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.974371 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.974372 (+ 1us) write_transaction.cc:95] Timestamp: L: 6461 1103 05:10:13.975139 (+ 767us) write_transaction.cc:102] APPLY: Starting 1103 05:10:13.975142 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975144 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975147 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975149 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975165 (+ 16us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975168 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975170 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975173 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975175 (+ 2us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975178 (+ 3us) tablet_metrics.cc:133] ProbeStats: blooms_consulted=0,keys_consulted=0,deltas_consulted=0,mrs_consulted=0 1103 05:10:13.975180 (+ 2us) write_transaction.cc:140] PRECOMMIT: Releasing row and schema locks 1103 05:10:13.9751 W1103 05:10:13.999795 7586 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 1042) took 107ms (client timeout 50). W1103 05:10:13.999929 7586 inbound_call.cc:140] Trace: 1103 05:10:13.892606 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:13.941301 (+ 48695us) service_pool.cc:152] Handling call 1103 05:10:13.999545 (+ 58244us) raft_consensus.cc:579] Updating replica for 8 ops 1103 05:10:13.999549 (+ 4us) raft_consensus.cc:614] Triggering prepare for 5 ops 1103 05:10:13.999688 (+ 139us) log.cc:406] Serialized 2510 byte log entry 1103 05:10:13.999701 (+ 13us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3654 1103 05:10:13.999706 (+ 5us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3662 1103 05:10:13.999707 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:13.999770 (+ 63us) raft_consensus.cc:697] finished 1103 05:10:13.999772 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:13.999775 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:10:13.999783 (+ 8us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:13.999618 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.999625 (+ 7us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.999626 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.999627 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.999637 (+ 10us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.999651 (+ 14us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.999652 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.999654 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.999657 (+ 3us) write_transaction.cc:95] Timestamp: L: 6420 Related trace: 1103 05:10:13.999662 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.999665 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.999666 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.999667 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.999674 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.999765 (+ 91us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.999766 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.999769 (+ 3us) write_transaction.cc:93] Start() 1103 05:10:13.999772 (+ 3us) write_transaction.cc:95] Timestamp: L: 6421 Related trace: 1103 05:10:13.999774 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.999779 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.999780 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.999781 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.999787 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.999795 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.999796 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.999798 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:13.999800 (+ 2us) write_transaction.cc:95] Timestamp: L: 6422 Related trace: 1103 05:10:13.999802 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.999805 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.999806 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.999806 (+ 0us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.999811 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.999818 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.999819 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.999820 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.999822 (+ 2us) write_transaction.cc:95] Timestamp: L: 6423 Related trace: 1103 05:10:13.999824 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:13.999826 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:13.999827 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:13.999828 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:13.999833 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:13.999841 (+ 8us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:13.999842 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:13.999843 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:13.999844 (+ 1us) write_transaction.cc:95] Timestamp: L: 6424 I1103 05:10:14.001057 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 36724.9 usec... I1103 05:10:14.007511 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 2335.21 usec... I1103 05:10:14.038171 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 28852.6 usec... I1103 05:10:14.042543 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 23124.3 usec... I1103 05:10:14.066061 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12851.6 usec... I1103 05:10:14.067443 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 28444.3 usec... I1103 05:10:14.085712 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 4156.17 usec... I1103 05:10:14.126684 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 11697 usec... I1103 05:10:14.127825 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 5683.87 usec... I1103 05:10:14.133283 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 68814.3 usec... I1103 05:10:14.138594 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 9957.44 usec... I1103 05:10:14.177878 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 16030 usec... I1103 05:10:14.202749 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 37277.7 usec... I1103 05:10:14.227033 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 44130.7 usec... I1103 05:10:14.240247 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 37049.3 usec... W1103 05:10:14.247402 31061 thread.cc:320] Waited for 9000ms trying to join with ts-test0 (tid 8513) I1103 05:10:14.271312 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 28282.9 usec... I1103 05:10:14.272126 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 6693.28 usec... I1103 05:10:14.277691 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 10141.2 usec... I1103 05:10:14.279028 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 46698.1 usec... I1103 05:10:14.300854 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 54704.6 usec... I1103 05:10:14.306778 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 30311.4 usec... I1103 05:10:14.361773 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 17430.9 usec... I1103 05:10:14.362529 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 13632.2 usec... I1103 05:10:14.376381 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 24545.6 usec... I1103 05:10:14.380017 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 16739.8 usec... I1103 05:10:14.391002 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12046.8 usec... I1103 05:10:14.396981 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 18309.4 usec... I1103 05:10:14.404131 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 1038.75 usec... I1103 05:10:14.436604 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 17713.9 usec... I1103 05:10:14.484184 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 22603.9 usec... I1103 05:10:14.493286 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 11467.4 usec... I1103 05:10:14.508147 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 29868.4 usec... I1103 05:10:14.508832 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 29535.1 usec... I1103 05:10:14.519572 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 47712 usec... W1103 05:10:14.558825 7710 consensus_peers.cc:368] Couldn't send request to peer 1135cdf4560a4f92ba4d3cb0c17f4eff for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3930-1.3930] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:14.567741 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 22118.7 usec... I1103 05:10:14.576498 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 6298.54 usec... I1103 05:10:14.583015 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 37547.3 usec... W1103 05:10:14.589474 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3931-1.3933] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:14.595849 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 35353 usec... I1103 05:10:14.611994 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 8244.98 usec... I1103 05:10:14.620856 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 2840.45 usec... W1103 05:10:14.621079 8085 log.cc:507] Time spent Append to log took a long time: real 0.083s user 0.000s sys 0.000s W1103 05:10:14.621186 7579 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:47919 (request call id 1128) took 113ms (client timeout 50). W1103 05:10:14.621251 7579 inbound_call.cc:140] Trace: 1103 05:10:14.507841 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:14.507861 (+ 20us) service_pool.cc:152] Handling call 1103 05:10:14.507872 (+ 11us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:14.507873 (+ 1us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:14.507898 (+ 25us) log.cc:406] Serialized 506 byte log entry 1103 05:10:14.507917 (+ 19us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3929 1103 05:10:14.507943 (+ 26us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3930 1103 05:10:14.507945 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:14.621164 (+113219us) raft_consensus.cc:697] finished 1103 05:10:14.621166 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:14.621168 (+ 2us) raft_consensus.cc:443] Updating watermarks 1103 05:10:14.621173 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:14.507907 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.507913 (+ 6us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.507915 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.507916 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.507924 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.507936 (+ 12us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.507937 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.507939 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:14.507943 (+ 4us) write_transaction.cc:95] Timestamp: L: 6871 I1103 05:10:14.624018 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 20946.9 usec... I1103 05:10:14.631424 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 12204.4 usec... W1103 05:10:14.643940 8084 log.cc:507] Time spent Append to log took a long time: real 0.053s user 0.000s sys 0.000s W1103 05:10:14.644007 7783 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 808) took 76ms (client timeout 50). W1103 05:10:14.644079 7783 inbound_call.cc:140] Trace: 1103 05:10:14.567489 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:14.567729 (+ 240us) service_pool.cc:152] Handling call 1103 05:10:14.590387 (+ 22658us) raft_consensus.cc:579] Updating replica for 3 ops 1103 05:10:14.590389 (+ 2us) raft_consensus.cc:614] Triggering prepare for 3 ops 1103 05:10:14.590431 (+ 42us) log.cc:406] Serialized 1518 byte log entry 1103 05:10:14.590437 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3930 1103 05:10:14.590447 (+ 10us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3933 1103 05:10:14.590448 (+ 1us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:14.643992 (+ 53544us) raft_consensus.cc:697] finished 1103 05:10:14.643994 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:14.643995 (+ 1us) raft_consensus.cc:443] Updating watermarks 1103 05:10:14.644000 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:14.590413 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.590418 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.590419 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.590420 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.590427 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.590437 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.590438 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.590439 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:14.590442 (+ 3us) write_transaction.cc:95] Timestamp: L: 7038 Related trace: 1103 05:10:14.590444 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.590447 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.590448 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.590449 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.590453 (+ 4us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.590459 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.590460 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.590461 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:14.590463 (+ 2us) write_transaction.cc:95] Timestamp: L: 7039 Related trace: 1103 05:10:14.590465 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.590467 (+ 2us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.590468 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.590469 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.590474 (+ 5us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.590479 (+ 5us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.590480 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.590481 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:14.590483 (+ 2us) write_transaction.cc:95] Timestamp: L: 7040 I1103 05:10:14.645258 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 40707 usec... I1103 05:10:14.663633 8541 raft_consensus-itest.cc:429] Delay injector thread for TS 1135cdf4560a4f92ba4d3cb0c17f4eff SIGSTOPped the ts, sleeping for 26591.2 usec... I1103 05:10:14.679726 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 8039.4 usec... W1103 05:10:14.686233 8093 log.cc:507] Time spent Append to log took a long time: real 0.147s user 0.000s sys 0.000s I1103 05:10:14.688160 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 21831.4 usec... I1103 05:10:14.694782 8549 raft_consensus-itest.cc:429] Delay injector thread for TS 20535f6222dc4aab9c9d9b5858d00404 SIGSTOPped the ts, sleeping for 29278.2 usec... W1103 05:10:14.726101 7785 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 809) took 38ms (client timeout 50). W1103 05:10:14.726202 7785 inbound_call.cc:140] Trace: 1103 05:10:14.688024 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:14.688041 (+ 17us) service_pool.cc:152] Handling call 1103 05:10:14.688076 (+ 35us) raft_consensus.cc:579] Updating replica for 8 ops 1103 05:10:14.688079 (+ 3us) raft_consensus.cc:614] Triggering prepare for 5 ops 1103 05:10:14.688136 (+ 57us) log.cc:406] Serialized 2510 byte log entry 1103 05:10:14.688143 (+ 7us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3930 1103 05:10:14.688145 (+ 2us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3938 1103 05:10:14.688147 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:14.726083 (+ 37936us) raft_consensus.cc:697] finished 1103 05:10:14.726085 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:14.726086 (+ 1us) raft_consensus.cc:443] Updating watermarks 1103 05:10:14.726091 (+ 5us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:14.688156 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.688161 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.688162 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.688163 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.688171 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.688180 (+ 9us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.688182 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.688183 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:14.688187 (+ 4us) write_transaction.cc:95] Timestamp: L: 7042 Related trace: 1103 05:10:14.688189 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.688192 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.688194 (+ 2us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.688197 (+ 3us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.710202 (+ 22005us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.710217 (+ 15us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.710218 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.710221 (+ 3us) write_transaction.cc:93] Start() 1103 05:10:14.710224 (+ 3us) write_transaction.cc:95] Timestamp: L: 7043 Related trace: 1103 05:10:14.710227 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.710232 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.710233 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.710235 (+ 2us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.710243 (+ 8us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.710253 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.710255 (+ 2us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.710257 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:14.710259 (+ 2us) write_transaction.cc:95] Timestamp: L: 7044 Related trace: 1103 05:10:14.710261 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.710265 (+ 4us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.710266 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.710267 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.710274 (+ 7us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.710281 (+ 7us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.710282 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.710284 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:14.710286 (+ 2us) write_transaction.cc:95] Timestamp: L: 7045 Related trace: 1103 05:10:14.710287 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.710290 (+ 3us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.710291 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.710291 (+ 0us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.710297 (+ 6us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.710303 (+ 6us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.710304 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.710305 (+ 1us) write_transaction.cc:93] Start() 1103 05:10:14.710307 (+ 2us) write_transaction.cc:95] Timestamp: L: 7046 I1103 05:10:14.728170 8557 raft_consensus-itest.cc:429] Delay injector thread for TS e95cd7e351fc4e28bac7f7977928f571 SIGSTOPped the ts, sleeping for 28479 usec... W1103 05:10:14.757232 8346 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:54764, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:14.777533 7712 consensus_peers.cc:368] Couldn't send request to peer e95cd7e351fc4e28bac7f7977928f571 for tablet 69ce28a55ab34dafb931a4677a97f8ae ops [1.3954-1.3954] Status: Timed out: Call timed out. Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:14.823839 8084 log.cc:507] Time spent Append to log took a long time: real 0.067s user 0.000s sys 0.000s W1103 05:10:14.823927 7782 inbound_call.cc:136] Call UpdateConsensus from 127.0.0.1:53374 (request call id 811) took 96ms (client timeout 50). W1103 05:10:14.823973 7782 inbound_call.cc:140] Trace: 1103 05:10:14.727576 (+ 0us) service_pool.cc:91] Inserting onto call queue 1103 05:10:14.727595 (+ 19us) service_pool.cc:152] Handling call 1103 05:10:14.727608 (+ 13us) raft_consensus.cc:579] Updating replica for 1 ops 1103 05:10:14.727611 (+ 3us) raft_consensus.cc:614] Triggering prepare for 1 ops 1103 05:10:14.727636 (+ 25us) log.cc:406] Serialized 496 byte log entry 1103 05:10:14.727642 (+ 6us) raft_consensus.cc:669] Marking committed up to term: 1 index: 3953 1103 05:10:14.727650 (+ 8us) raft_consensus.cc:682] Updating last received op as term: 1 index: 3954 1103 05:10:14.727652 (+ 2us) raft_consensus.cc:695] Waiting on the replicates to finish logging 1103 05:10:14.823903 (+ 96251us) raft_consensus.cc:697] finished 1103 05:10:14.823905 (+ 2us) raft_consensus.cc:710] UpdateReplicas() finished 1103 05:10:14.823908 (+ 3us) raft_consensus.cc:443] Updating watermarks 1103 05:10:14.823915 (+ 7us) inbound_call.cc:83] Queueing success response Related trace: 1103 05:10:14.728043 (+ 0us) write_transaction.cc:62] PREPARE: Starting 1103 05:10:14.728048 (+ 5us) write_transaction.cc:221] Acquiring schema lock in shared mode 1103 05:10:14.728049 (+ 1us) write_transaction.cc:223] Acquired schema lock 1103 05:10:14.728050 (+ 1us) tablet.cc:207] PREPARE: Decoding operations 1103 05:10:14.728060 (+ 10us) tablet.cc:231] PREPARE: Acquiring locks for 10 operations 1103 05:10:14.728070 (+ 10us) tablet.cc:236] PREPARE: locks acquired 1103 05:10:14.728071 (+ 1us) write_transaction.cc:88] PREPARE: finished. 1103 05:10:14.728073 (+ 2us) write_transaction.cc:93] Start() 1103 05:10:14.728076 (+ 3us) write_transaction.cc:95] Timestamp: L: 7062 W1103 05:10:15.159040 8349 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58392, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:15.594503 8346 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:54764, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:15.931385 8349 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58392, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:16.346887 8347 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:42739, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:16.762109 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master with pid 7493 I1103 05:10:16.764962 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 7545 I1103 05:10:16.829686 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 7595 I1103 05:10:16.835479 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 7750 [ OK ] DistConsensusTest.MultiThreadedMutateAndInsertThroughConsensus (15783 ms) [ RUN ] DistConsensusTest.TestInsertOnNonLeader I1103 05:10:16.845811 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master kudu-master --master_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --master_web_port=0 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:16.893733 9980 master_main.cc:27] Initializing master server... I1103 05:10:16.959460 9980 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/master/instance: uuid: "33dfbeed556e477cb4452c7f3a1a5bb8" format_stamp: "Formatted at 2014-11-03 13:10:16 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:16.959588 9980 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/master uuid: "33dfbeed556e477cb4452c7f3a1a5bb8" format_stamp: "Formatted at 2014-11-03 13:10:16 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:16.962327 9980 master_main.cc:30] Starting master server... I1103 05:10:16.964843 9980 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:16.964877 9980 webserver.cc:117] Document root disabled I1103 05:10:16.965373 9980 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:39443 I1103 05:10:16.965565 9980 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb I1103 05:10:16.975893 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:33817 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:17.021322 10098 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:17.066221 10098 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/instance: uuid: "22c425bf11c64949a0491bb29a07a270" format_stamp: "Formatted at 2014-11-03 13:10:17 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:17.066366 10098 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 uuid: "22c425bf11c64949a0491bb29a07a270" format_stamp: "Formatted at 2014-11-03 13:10:17 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:17.069483 10098 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:17.072631 10098 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:17.072661 10098 webserver.cc:117] Document root disabled I1103 05:10:17.073125 10098 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:41976 I1103 05:10:17.073315 10098 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb I1103 05:10:17.073431 10098 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:17.073532 10098 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:17.074888 10148 heartbeater.cc:227] Registering TS with master... I1103 05:10:17.074909 10148 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:17.076599 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:33817 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:17.119576 10152 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:17.170174 10152 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/instance: uuid: "e99b7680214b4567ae040ad86b2be249" format_stamp: "Formatted at 2014-11-03 13:10:17 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:17.170285 10152 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 uuid: "e99b7680214b4567ae040ad86b2be249" format_stamp: "Formatted at 2014-11-03 13:10:17 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:17.173213 10152 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:17.176249 10152 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:17.176280 10152 webserver.cc:117] Document root disabled I1103 05:10:17.176759 10152 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:44678 I1103 05:10:17.176939 10152 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb I1103 05:10:17.177067 10152 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:17.177242 10152 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:17.178246 10290 heartbeater.cc:227] Registering TS with master... I1103 05:10:17.178282 10290 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:17.179414 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:33817 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=50 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:17.228446 10309 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:17.426520 10309 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/instance: uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" format_stamp: "Formatted at 2014-11-03 13:10:17 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:17.426648 10309 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" format_stamp: "Formatted at 2014-11-03 13:10:17 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:17.426676 10053 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: Bootstrap starting. I1103 05:10:17.427057 10053 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one. I1103 05:10:17.427321 10053 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:17.427486 10314 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:17.427928 10053 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: No bootstrap required, opened a new log I1103 05:10:17.428966 10315 tablet_peer.cc:187] Configuration changed for tablet: 11111111111111111111111111111111 in peer: 33dfbeed556e477cb4452c7f3a1a5bb8 Changed from: seqno: 0 local: true peers { permanent_uuid: "33dfbeed556e477cb4452c7f3a1a5bb8" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "33dfbeed556e477cb4452c7f3a1a5bb8" role: LEADER } I1103 05:10:17.429013 10315 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "33dfbeed556e477cb4452c7f3a1a5bb8" role: LEADER } I1103 05:10:17.429949 10309 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:17.433059 10309 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:17.433089 10309 webserver.cc:117] Document root disabled I1103 05:10:17.433569 10309 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:38084 I1103 05:10:17.433763 10309 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertOnNonLeader.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb I1103 05:10:17.433887 10309 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:17.433994 10309 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:17.435086 10357 heartbeater.cc:227] Registering TS with master... I1103 05:10:17.435107 10357 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:17.836575 10053 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: Bootstrap starting. I1103 05:10:17.836937 10053 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I1103 05:10:17.837771 10053 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I1103 05:10:17.838146 10053 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. W1103 05:10:17.838193 10032 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:10:17.838217 10034 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:10:17.838371 10036 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. I1103 05:10:17.838515 10384 tablet_peer.cc:187] Configuration changed for tablet: 00000000000000000000000000000000 in peer: 33dfbeed556e477cb4452c7f3a1a5bb8 Changed from: seqno: 0 local: true peers { permanent_uuid: "33dfbeed556e477cb4452c7f3a1a5bb8" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "33dfbeed556e477cb4452c7f3a1a5bb8" role: LEADER } I1103 05:10:17.838557 10384 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "33dfbeed556e477cb4452c7f3a1a5bb8" role: LEADER } I1103 05:10:17.838568 10053 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:10:17.838706 9980 master_main.cc:33] Master server successfully started. I1103 05:10:18.838857 10040 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" instance_seqno: 1415020217176109 } as {real_user=jenkins, eff_user=} at 127.0.0.1:56936; Asking this server to re-register. I1103 05:10:18.838866 10039 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" instance_seqno: 1415020217072498 } as {real_user=jenkins, eff_user=} at 127.0.0.1:56911; Asking this server to re-register. I1103 05:10:18.839097 10031 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" instance_seqno: 1415020217432923 } as {real_user=jenkins, eff_user=} at 127.0.0.1:56940; Asking this server to re-register. I1103 05:10:18.839282 10148 heartbeater.cc:227] Registering TS with master... I1103 05:10:18.839315 10148 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:18.839341 10290 heartbeater.cc:227] Registering TS with master... I1103 05:10:18.839372 10290 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:18.839491 10357 heartbeater.cc:227] Registering TS with master... I1103 05:10:18.839522 10357 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:18.839767 10035 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" instance_seqno: 1415020217176109 } with Master I1103 05:10:18.839823 10037 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" instance_seqno: 1415020217072498 } with Master I1103 05:10:18.839923 10034 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" instance_seqno: 1415020217432923 } with Master I1103 05:10:18.840759 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:10:18.841094 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:10:18.843579 10040 catalog_manager.cc:437] CreateTable from {real_user=jenkins, eff_user=} at 127.0.0.1:56946: name: "TestTable" schema { columns { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I1103 05:10:18.845696 10335 tablet_service.cc:264] Processing CreateTablet for tablet 274a13a24ea74ab685709e43dfd875a1 (table=TestTable [id=bcd16f01952d4bd18887192ee34eea16]), range=["", ""] I1103 05:10:18.846009 10126 tablet_service.cc:264] Processing CreateTablet for tablet 274a13a24ea74ab685709e43dfd875a1 (table=TestTable [id=bcd16f01952d4bd18887192ee34eea16]), range=["", ""] I1103 05:10:18.846101 10268 tablet_service.cc:264] Processing CreateTablet for tablet 274a13a24ea74ab685709e43dfd875a1 (table=TestTable [id=bcd16f01952d4bd18887192ee34eea16]), range=["", ""] I1103 05:10:19.223881 10126 ts_tablet_manager.cc:464] Registered tablet 274a13a24ea74ab685709e43dfd875a1 I1103 05:10:19.224015 10617 ts_tablet_manager.cc:325] Bootstrapping tablet: 274a13a24ea74ab685709e43dfd875a1 I1103 05:10:19.224140 10617 tablet_bootstrap.cc:234] Tablet 274a13a24ea74ab685709e43dfd875a1: Bootstrap starting. I1103 05:10:19.224241 10335 ts_tablet_manager.cc:464] Registered tablet 274a13a24ea74ab685709e43dfd875a1 I1103 05:10:19.224365 10618 ts_tablet_manager.cc:325] Bootstrapping tablet: 274a13a24ea74ab685709e43dfd875a1 I1103 05:10:19.224520 10618 tablet_bootstrap.cc:234] Tablet 274a13a24ea74ab685709e43dfd875a1: Bootstrap starting. I1103 05:10:19.224597 10617 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 274a13a24ea74ab685709e43dfd875a1 creating new one. I1103 05:10:19.224848 10617 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:10:19.224941 10618 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 274a13a24ea74ab685709e43dfd875a1 creating new one. W1103 05:10:19.225028 10621 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:19.225219 10618 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:10:19.225414 10617 tablet_bootstrap.cc:234] Tablet 274a13a24ea74ab685709e43dfd875a1: No bootstrap required, opened a new log W1103 05:10:19.225461 10624 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:19.225476 10617 ts_tablet_manager.cc:330] Time spent tablet 274a13a24ea74ab685709e43dfd875a1 bootstrap: real 0.001s user 0.001s sys 0.000s I1103 05:10:19.225836 10617 raft_consensus.cc:102] T 274a13a24ea74ab685709e43dfd875a1 P 22c425bf11c64949a0491bb29a07a270 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:19.225858 10618 tablet_bootstrap.cc:234] Tablet 274a13a24ea74ab685709e43dfd875a1: No bootstrap required, opened a new log I1103 05:10:19.225908 10618 ts_tablet_manager.cc:330] Time spent tablet 274a13a24ea74ab685709e43dfd875a1 bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:10:19.225980 10617 raft_consensus.cc:302] T 274a13a24ea74ab685709e43dfd875a1 P 22c425bf11c64949a0491bb29a07a270 [FOLLOWER]: Becoming Follower/Learner I1103 05:10:19.226040 10617 ts_tablet_manager.cc:350] Time spent starting tablet 274a13a24ea74ab685709e43dfd875a1: real 0.001s user 0.000s sys 0.000s I1103 05:10:19.226238 10618 raft_consensus.cc:102] T 274a13a24ea74ab685709e43dfd875a1 P 69a1dcdbcc5c4812825c7be6386a1ca9 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:19.226354 10618 raft_consensus.cc:302] T 274a13a24ea74ab685709e43dfd875a1 P 69a1dcdbcc5c4812825c7be6386a1ca9 [FOLLOWER]: Becoming Follower/Learner I1103 05:10:19.226410 10618 ts_tablet_manager.cc:350] Time spent starting tablet 274a13a24ea74ab685709e43dfd875a1: real 0.000s user 0.000s sys 0.000s I1103 05:10:19.474789 10268 ts_tablet_manager.cc:464] Registered tablet 274a13a24ea74ab685709e43dfd875a1 I1103 05:10:19.474884 10632 ts_tablet_manager.cc:325] Bootstrapping tablet: 274a13a24ea74ab685709e43dfd875a1 I1103 05:10:19.475033 10632 tablet_bootstrap.cc:234] Tablet 274a13a24ea74ab685709e43dfd875a1: Bootstrap starting. I1103 05:10:19.475539 10632 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 274a13a24ea74ab685709e43dfd875a1 creating new one. I1103 05:10:19.475857 10632 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:19.476256 10633 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:19.476524 10632 tablet_bootstrap.cc:234] Tablet 274a13a24ea74ab685709e43dfd875a1: No bootstrap required, opened a new log I1103 05:10:19.476584 10632 ts_tablet_manager.cc:330] Time spent tablet 274a13a24ea74ab685709e43dfd875a1 bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:10:19.477016 10632 raft_consensus.cc:102] T 274a13a24ea74ab685709e43dfd875a1 P e99b7680214b4567ae040ad86b2be249 [CANDIDATE]: Replica starting. Triggering 0 pending transactions. I1103 05:10:19.477391 10632 raft_consensus.cc:235] T 274a13a24ea74ab685709e43dfd875a1 P e99b7680214b4567ae040ad86b2be249 [LEADER]: Becoming Leader I1103 05:10:19.479730 10632 ts_tablet_manager.cc:350] Time spent starting tablet 274a13a24ea74ab685709e43dfd875a1: real 0.003s user 0.001s sys 0.001s I1103 05:10:19.905916 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master W1103 05:10:19.906038 9934 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:33817, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:19.906743 31061 raft_consensus-itest.cc:232] Couldn't find the leader and/or replicas. Locations: tablet_id: "274a13a24ea74ab685709e43dfd875a1" start_key: "" end_key: "" replicas { ts_info { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" rpc_addresses { host: "127.0.0.1" port: 58699 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" rpc_addresses { host: "127.0.0.1" port: 49288 } } role: CANDIDATE } replicas { ts_info { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" rpc_addresses { host: "127.0.0.1" port: 55959 } } role: FOLLOWER } stale: false W1103 05:10:19.978332 10637 raft_consensus_state.cc:463] T 274a13a24ea74ab685709e43dfd875a1 P e99b7680214b4567ae040ad86b2be249 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 W1103 05:10:19.978590 10634 raft_consensus_state.cc:463] T 274a13a24ea74ab685709e43dfd875a1 P e99b7680214b4567ae040ad86b2be249 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:10:19.979917 10345 raft_consensus.cc:608] T 274a13a24ea74ab685709e43dfd875a1 P 69a1dcdbcc5c4812825c7be6386a1ca9 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:19.980038 10253 consensus_queue.cc:451] Connected to peer: 69a1dcdbcc5c4812825c7be6386a1ca9 whose last received watermark is: term: 0 index: 0 I1103 05:10:19.980154 10136 raft_consensus.cc:608] T 274a13a24ea74ab685709e43dfd875a1 P 22c425bf11c64949a0491bb29a07a270 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:19.980304 10252 consensus_queue.cc:451] Connected to peer: 22c425bf11c64949a0491bb29a07a270 whose last received watermark is: term: 0 index: 0 I1103 05:10:19.980700 10253 raft_consensus_state.cc:456] T 274a13a24ea74ab685709e43dfd875a1 P e99b7680214b4567ae040ad86b2be249 [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I1103 05:10:19.980820 10634 tablet_peer.cc:187] Configuration changed for tablet: 274a13a24ea74ab685709e43dfd875a1 in peer: e99b7680214b4567ae040ad86b2be249 Changed from: seqno: -1 local: false peers { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55959 } } peers { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58699 } } peers { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 49288 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55959 } } peers { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58699 } } peers { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" role: LEADER last_known_addr { host: "127.0.0.1" port: 49288 } } I1103 05:10:20.480906 10625 tablet_peer.cc:187] Configuration changed for tablet: 274a13a24ea74ab685709e43dfd875a1 in peer: 69a1dcdbcc5c4812825c7be6386a1ca9 Changed from: seqno: -1 local: false peers { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55959 } } peers { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58699 } } peers { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 49288 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55959 } } peers { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58699 } } peers { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" role: LEADER last_known_addr { host: "127.0.0.1" port: 49288 } } I1103 05:10:20.480976 10623 tablet_peer.cc:187] Configuration changed for tablet: 274a13a24ea74ab685709e43dfd875a1 in peer: 22c425bf11c64949a0491bb29a07a270 Changed from: seqno: -1 local: false peers { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55959 } } peers { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58699 } } peers { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 49288 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55959 } } peers { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58699 } } peers { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" role: LEADER last_known_addr { host: "127.0.0.1" port: 49288 } } I1103 05:10:20.841270 10033 catalog_manager.cc:1127] Tablet: 274a13a24ea74ab685709e43dfd875a1 reported quorum change. New quorum: seqno: 0 local: false peers { permanent_uuid: "69a1dcdbcc5c4812825c7be6386a1ca9" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 55959 } } peers { permanent_uuid: "22c425bf11c64949a0491bb29a07a270" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 58699 } } peers { permanent_uuid: "e99b7680214b4567ae040ad86b2be249" role: LEADER last_known_addr { host: "127.0.0.1" port: 49288 } } W1103 05:10:20.907009 9934 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:33817, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:20.907516 10704 reactor.cc:199] Client call Write has no timeout set for connection id: {remote=127.0.0.1:55959, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:20.909052 10706 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:49288, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:20.910547 10704 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:55959, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:20.911228 10703 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58699, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:20.912467 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master with pid 9980 I1103 05:10:20.914803 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 10098 I1103 05:10:20.917954 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 10152 I1103 05:10:20.920326 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 10309 [ OK ] DistConsensusTest.TestInsertOnNonLeader (4079 ms) [ RUN ] DistConsensusTest.TestRunLeaderElection I1103 05:10:20.925124 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master kudu-master --master_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --master_web_port=0 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:20.963664 10805 master_main.cc:27] Initializing master server... I1103 05:10:22.693719 10805 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/master/instance: uuid: "e15743e460474612a325d84867b1b4f5" format_stamp: "Formatted at 2014-11-03 13:10:20 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:24.830348 10805 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/master uuid: "e15743e460474612a325d84867b1b4f5" format_stamp: "Formatted at 2014-11-03 13:10:20 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:24.833113 10805 master_main.cc:30] Starting master server... I1103 05:10:24.835494 10805 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:24.835536 10805 webserver.cc:117] Document root disabled I1103 05:10:24.836033 10805 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:41429 I1103 05:10:24.842859 10805 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb I1103 05:10:24.845201 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:46911 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:24.887768 10890 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:24.921608 10890 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/instance: uuid: "528ee71f35ff459a88a919665f288e3e" format_stamp: "Formatted at 2014-11-03 13:10:24 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:24.921728 10890 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 uuid: "528ee71f35ff459a88a919665f288e3e" format_stamp: "Formatted at 2014-11-03 13:10:24 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:24.925602 10890 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:24.929838 10890 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:24.929878 10890 webserver.cc:117] Document root disabled I1103 05:10:24.930485 10890 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:58533 I1103 05:10:24.930738 10890 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb I1103 05:10:24.930892 10890 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:24.931054 10890 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:24.932679 10991 heartbeater.cc:227] Registering TS with master... I1103 05:10:24.932708 10991 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:24.939263 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:46911 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:24.945243 10848 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: Bootstrap starting. I1103 05:10:24.945726 10848 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one. I1103 05:10:24.946020 10848 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:24.946179 10999 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:24.947281 10848 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: No bootstrap required, opened a new log I1103 05:10:24.951715 11000 tablet_peer.cc:187] Configuration changed for tablet: 11111111111111111111111111111111 in peer: e15743e460474612a325d84867b1b4f5 Changed from: seqno: 0 local: true peers { permanent_uuid: "e15743e460474612a325d84867b1b4f5" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "e15743e460474612a325d84867b1b4f5" role: LEADER } I1103 05:10:24.951760 11000 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "e15743e460474612a325d84867b1b4f5" role: LEADER } I1103 05:10:24.994503 11041 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:25.028622 11041 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/instance: uuid: "b27fcd8b4a624b41a7ead590b376399d" format_stamp: "Formatted at 2014-11-03 13:10:24 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:25.028790 11041 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 uuid: "b27fcd8b4a624b41a7ead590b376399d" format_stamp: "Formatted at 2014-11-03 13:10:24 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:25.047199 11041 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:25.055834 11041 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:25.055871 11041 webserver.cc:117] Document root disabled I1103 05:10:25.056464 11041 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:45622 I1103 05:10:25.056757 11041 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb I1103 05:10:25.056921 11041 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:25.057051 11041 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:25.059347 11116 heartbeater.cc:227] Registering TS with master... I1103 05:10:25.059392 11116 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:25.067034 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:46911 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:25.126194 10848 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: Bootstrap starting. I1103 05:10:25.126631 10848 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I1103 05:10:25.127275 10848 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I1103 05:10:25.127723 10848 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. W1103 05:10:25.127789 10825 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. W1103 05:10:25.127818 10823 master_service.cc:103] Catalog manager is not yet initialized, ignoring the heartbeat. I1103 05:10:25.128346 11166 tablet_peer.cc:187] Configuration changed for tablet: 00000000000000000000000000000000 in peer: e15743e460474612a325d84867b1b4f5 Changed from: seqno: 0 local: true peers { permanent_uuid: "e15743e460474612a325d84867b1b4f5" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "e15743e460474612a325d84867b1b4f5" role: LEADER } I1103 05:10:25.128388 11166 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "e15743e460474612a325d84867b1b4f5" role: LEADER } I1103 05:10:25.128394 10848 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:10:25.128521 10805 master_main.cc:33] Master server successfully started. I1103 05:10:25.138799 11153 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:25.154523 11153 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/instance: uuid: "7853062a1e224f5ebc51a9285545e696" format_stamp: "Formatted at 2014-11-03 13:10:25 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:25.154682 11153 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 uuid: "7853062a1e224f5ebc51a9285545e696" format_stamp: "Formatted at 2014-11-03 13:10:25 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:25.177198 11153 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:25.184552 11153 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:25.184571 11153 webserver.cc:117] Document root disabled I1103 05:10:25.185081 11153 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:35024 I1103 05:10:25.186511 11153 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb I1103 05:10:25.195843 11153 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:25.200098 11227 heartbeater.cc:227] Registering TS with master... I1103 05:10:25.200129 11227 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:25.200660 10824 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" instance_seqno: 1415020225184348 } with Master I1103 05:10:25.204224 11153 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:26.128450 10826 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" instance_seqno: 1415020225055620 } as {real_user=jenkins, eff_user=} at 127.0.0.1:33719; Asking this server to re-register. I1103 05:10:26.128525 10826 master_service.cc:129] Got heartbeat from unknown tablet server { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" instance_seqno: 1415020224929657 } as {real_user=jenkins, eff_user=} at 127.0.0.1:33717; Asking this server to re-register. I1103 05:10:26.128777 11116 heartbeater.cc:227] Registering TS with master... I1103 05:10:26.128813 11116 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:26.128856 10991 heartbeater.cc:227] Registering TS with master... I1103 05:10:26.128886 10991 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:26.129050 10830 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" instance_seqno: 1415020224929657 } with Master I1103 05:10:26.129112 10830 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" instance_seqno: 1415020225055620 } with Master I1103 05:10:26.130100 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:10:26.130503 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:10:26.157814 10825 catalog_manager.cc:437] CreateTable from {real_user=jenkins, eff_user=} at 127.0.0.1:33728: name: "TestTable" schema { columns { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I1103 05:10:26.186350 10969 tablet_service.cc:264] Processing CreateTablet for tablet 805d0620dd984789a89c70efd481e29c (table=TestTable [id=293487f3c5cb4d51b23379aa7c6e5fdb]), range=["", ""] I1103 05:10:26.187181 11088 tablet_service.cc:264] Processing CreateTablet for tablet 805d0620dd984789a89c70efd481e29c (table=TestTable [id=293487f3c5cb4d51b23379aa7c6e5fdb]), range=["", ""] I1103 05:10:26.187726 11202 tablet_service.cc:264] Processing CreateTablet for tablet 805d0620dd984789a89c70efd481e29c (table=TestTable [id=293487f3c5cb4d51b23379aa7c6e5fdb]), range=["", ""] I1103 05:10:30.476214 11202 ts_tablet_manager.cc:464] Registered tablet 805d0620dd984789a89c70efd481e29c I1103 05:10:30.476301 11756 ts_tablet_manager.cc:325] Bootstrapping tablet: 805d0620dd984789a89c70efd481e29c I1103 05:10:30.476444 11756 tablet_bootstrap.cc:234] Tablet 805d0620dd984789a89c70efd481e29c: Bootstrap starting. I1103 05:10:30.476832 11756 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 805d0620dd984789a89c70efd481e29c creating new one. I1103 05:10:30.477103 11756 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:30.477444 11757 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:30.477656 11756 tablet_bootstrap.cc:234] Tablet 805d0620dd984789a89c70efd481e29c: No bootstrap required, opened a new log I1103 05:10:30.477710 11756 ts_tablet_manager.cc:330] Time spent tablet 805d0620dd984789a89c70efd481e29c bootstrap: real 0.001s user 0.001s sys 0.000s I1103 05:10:30.478070 11756 raft_consensus.cc:102] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:30.478206 11756 raft_consensus.cc:302] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [FOLLOWER]: Becoming Follower/Learner I1103 05:10:30.478266 11756 ts_tablet_manager.cc:350] Time spent starting tablet 805d0620dd984789a89c70efd481e29c: real 0.001s user 0.000s sys 0.000s I1103 05:10:30.481504 11088 ts_tablet_manager.cc:464] Registered tablet 805d0620dd984789a89c70efd481e29c I1103 05:10:30.481575 11759 ts_tablet_manager.cc:325] Bootstrapping tablet: 805d0620dd984789a89c70efd481e29c I1103 05:10:30.481689 11759 tablet_bootstrap.cc:234] Tablet 805d0620dd984789a89c70efd481e29c: Bootstrap starting. I1103 05:10:30.482074 11759 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 805d0620dd984789a89c70efd481e29c creating new one. I1103 05:10:30.482300 11759 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:30.482626 11760 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:30.482828 11759 tablet_bootstrap.cc:234] Tablet 805d0620dd984789a89c70efd481e29c: No bootstrap required, opened a new log I1103 05:10:30.482877 11759 ts_tablet_manager.cc:330] Time spent tablet 805d0620dd984789a89c70efd481e29c bootstrap: real 0.001s user 0.001s sys 0.000s I1103 05:10:30.483201 11759 raft_consensus.cc:102] T 805d0620dd984789a89c70efd481e29c P b27fcd8b4a624b41a7ead590b376399d [CANDIDATE]: Replica starting. Triggering 0 pending transactions. I1103 05:10:30.483479 11759 raft_consensus.cc:235] T 805d0620dd984789a89c70efd481e29c P b27fcd8b4a624b41a7ead590b376399d [LEADER]: Becoming Leader I1103 05:10:30.485169 10969 ts_tablet_manager.cc:464] Registered tablet 805d0620dd984789a89c70efd481e29c I1103 05:10:30.485244 11765 ts_tablet_manager.cc:325] Bootstrapping tablet: 805d0620dd984789a89c70efd481e29c I1103 05:10:30.485411 11765 tablet_bootstrap.cc:234] Tablet 805d0620dd984789a89c70efd481e29c: Bootstrap starting. I1103 05:10:30.485816 11765 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 805d0620dd984789a89c70efd481e29c creating new one. I1103 05:10:30.485867 11759 ts_tablet_manager.cc:350] Time spent starting tablet 805d0620dd984789a89c70efd481e29c: real 0.003s user 0.000s sys 0.001s I1103 05:10:30.486075 11765 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:30.486418 11767 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:30.486636 11765 tablet_bootstrap.cc:234] Tablet 805d0620dd984789a89c70efd481e29c: No bootstrap required, opened a new log I1103 05:10:30.486685 11765 ts_tablet_manager.cc:330] Time spent tablet 805d0620dd984789a89c70efd481e29c bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:10:30.487000 11765 raft_consensus.cc:102] T 805d0620dd984789a89c70efd481e29c P 528ee71f35ff459a88a919665f288e3e [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:30.487128 11765 raft_consensus.cc:302] T 805d0620dd984789a89c70efd481e29c P 528ee71f35ff459a88a919665f288e3e [FOLLOWER]: Becoming Follower/Learner I1103 05:10:30.487185 11765 ts_tablet_manager.cc:350] Time spent starting tablet 805d0620dd984789a89c70efd481e29c: real 0.000s user 0.000s sys 0.000s W1103 05:10:30.984890 11764 raft_consensus_state.cc:463] T 805d0620dd984789a89c70efd481e29c P b27fcd8b4a624b41a7ead590b376399d [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 W1103 05:10:30.985121 11761 raft_consensus_state.cc:463] T 805d0620dd984789a89c70efd481e29c P b27fcd8b4a624b41a7ead590b376399d [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:10:30.986316 10979 raft_consensus.cc:608] T 805d0620dd984789a89c70efd481e29c P 528ee71f35ff459a88a919665f288e3e [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:30.986440 11063 consensus_queue.cc:451] Connected to peer: 528ee71f35ff459a88a919665f288e3e whose last received watermark is: term: 0 index: 0 I1103 05:10:30.986655 11212 raft_consensus.cc:608] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:30.986793 11072 consensus_queue.cc:451] Connected to peer: 7853062a1e224f5ebc51a9285545e696 whose last received watermark is: term: 0 index: 0 I1103 05:10:30.987541 11063 raft_consensus_state.cc:456] T 805d0620dd984789a89c70efd481e29c P b27fcd8b4a624b41a7ead590b376399d [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I1103 05:10:30.987646 11761 tablet_peer.cc:187] Configuration changed for tablet: 805d0620dd984789a89c70efd481e29c in peer: b27fcd8b4a624b41a7ead590b376399d Changed from: seqno: -1 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 51676 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: LEADER last_known_addr { host: "127.0.0.1" port: 51676 } } I1103 05:10:31.131703 10831 catalog_manager.cc:1127] Tablet: 805d0620dd984789a89c70efd481e29c reported quorum change. New quorum: seqno: 0 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: LEADER last_known_addr { host: "127.0.0.1" port: 51676 } } /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/src/kudu/integration-tests/raft_consensus-itest.cc:136: Failure Failed Bad status: Timed out: Timeout out waiting for Table Creation I1103 05:10:31.224298 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master W1103 05:10:31.224413 10804 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:46911, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:31.236893 11768 tablet_peer.cc:187] Configuration changed for tablet: 805d0620dd984789a89c70efd481e29c in peer: 528ee71f35ff459a88a919665f288e3e Changed from: seqno: -1 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 51676 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: LEADER last_known_addr { host: "127.0.0.1" port: 51676 } } I1103 05:10:31.237334 11758 tablet_peer.cc:187] Configuration changed for tablet: 805d0620dd984789a89c70efd481e29c in peer: 7853062a1e224f5ebc51a9285545e696 Changed from: seqno: -1 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 51676 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: LEADER last_known_addr { host: "127.0.0.1" port: 51676 } } W1103 05:10:31.302281 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51676, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.343605 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.404409 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51676, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.443058 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.508009 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51676, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.545852 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.618402 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51676, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.655077 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.738855 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51676, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.775461 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.869720 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51676, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.906265 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:31.951527 11794 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51264, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:31.998714 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 11041 W1103 05:10:32.002939 11794 reactor.cc:199] Client call RunLeaderElection has no timeout set for connection id: {remote=127.0.0.1:51264, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:32.003710 11215 raft_consensus.cc:180] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [CANDIDATE]: Starting election with quorum: seqno: 1 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51676 } } I1103 05:10:32.003881 11215 leader_election.cc:183] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696: Term 2 election: Requesting vote from peer b27fcd8b4a624b41a7ead590b376399d I1103 05:10:32.003911 11215 leader_election.cc:183] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696: Term 2 election: Requesting vote from peer 528ee71f35ff459a88a919665f288e3e W1103 05:10:32.004061 11809 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.004202 11176 leader_election.cc:218] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696: Term 2 election: RPC error from VoteRequest() call to peer b27fcd8b4a624b41a7ead590b376399d: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) I1103 05:10:32.005255 10982 raft_consensus.cc:953] Tablet 805d0620dd984789a89c70efd481e29c: Replica 528ee71f35ff459a88a919665f288e3e: Leader election vote request: Granting yes vote for candidate 7853062a1e224f5ebc51a9285545e696 in term 2. I1103 05:10:32.005362 11176 leader_election.cc:300] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696: Term 2 election: Vote granted by peer 528ee71f35ff459a88a919665f288e3e I1103 05:10:32.005380 11176 leader_election.cc:273] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696: Term 2 election: Election decided. Result: candidate won. I1103 05:10:32.005398 11176 raft_consensus.cc:1029] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [CANDIDATE]: Leader election won for term 2 I1103 05:10:32.005424 11176 raft_consensus.cc:1046] Calling BecomeLeaderUnlocked()... I1103 05:10:32.005435 11176 raft_consensus.cc:235] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [LEADER]: Becoming Leader W1103 05:10:32.014792 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.014858 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.025866 11822 raft_consensus_state.cc:463] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 1 index: 51 New majority replicated is: term: 1 index: 51. Current term is: 2 W1103 05:10:32.025962 11811 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.025972 11758 raft_consensus_state.cc:463] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 1 index: 51 New majority replicated is: term: 0 index: 0. Current term is: 2 I1103 05:10:32.026003 10983 raft_consensus.cc:608] T 805d0620dd984789a89c70efd481e29c P 528ee71f35ff459a88a919665f288e3e [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 1 index: 51. Preceding OpId from leader: term: 2 index: 53. W1103 05:10:32.026026 11176 consensus_peers.cc:368] Couldn't send request to peer b27fcd8b4a624b41a7ead590b376399d for tablet 805d0620dd984789a89c70efd481e29c ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:32.026111 11176 consensus_queue.cc:451] Connected to peer: 528ee71f35ff459a88a919665f288e3e whose last received watermark is: term: 1 index: 51 I1103 05:10:32.026861 11176 raft_consensus_state.cc:456] T 805d0620dd984789a89c70efd481e29c P 7853062a1e224f5ebc51a9285545e696 [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 1 index: 51 New committed index is: term: 2 index: 53 I1103 05:10:32.026940 11758 tablet_peer.cc:187] Configuration changed for tablet: 805d0620dd984789a89c70efd481e29c in peer: 7853062a1e224f5ebc51a9285545e696 Changed from: seqno: 0 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: LEADER last_known_addr { host: "127.0.0.1" port: 51676 } } Changed to: seqno: 1 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: LEADER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51676 } } W1103 05:10:32.035218 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.035259 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) I1103 05:10:32.043725 11768 tablet_peer.cc:187] Configuration changed for tablet: 805d0620dd984789a89c70efd481e29c in peer: 528ee71f35ff459a88a919665f288e3e Changed from: seqno: 0 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: LEADER last_known_addr { host: "127.0.0.1" port: 51676 } } Changed to: seqno: 1 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: LEADER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51676 } } W1103 05:10:32.045608 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.045650 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.054471 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.054512 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.067859 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.067903 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.086993 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.087040 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.101250 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.101297 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.119058 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.119106 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.131192 11818 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.131240 11813 meta_cache.cc:159] Replica 805d0620dd984789a89c70efd481e29c on ts b27fcd8b4a624b41a7ead590b376399d has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) I1103 05:10:32.132843 10825 catalog_manager.cc:1127] Tablet: 805d0620dd984789a89c70efd481e29c reported quorum change. New quorum: seqno: 1 local: false peers { permanent_uuid: "528ee71f35ff459a88a919665f288e3e" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 44311 } } peers { permanent_uuid: "7853062a1e224f5ebc51a9285545e696" role: LEADER last_known_addr { host: "127.0.0.1" port: 51264 } } peers { permanent_uuid: "b27fcd8b4a624b41a7ead590b376399d" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 51676 } } W1103 05:10:32.196421 11794 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51264, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:32.275857 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:32.381957 11794 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51264, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:32.458492 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:32.506305 11809 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111) W1103 05:10:32.506350 11176 consensus_peers.cc:368] Couldn't send request to peer b27fcd8b4a624b41a7ead590b376399d for tablet 805d0620dd984789a89c70efd481e29c ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:51676: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1103 05:10:32.571419 11794 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51264, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:32.647934 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:32.771239 11794 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:51264, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:32.847400 11791 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:44311, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:32.942632 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master with pid 10805 I1103 05:10:32.944953 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 10890 I1103 05:10:32.949153 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 11153 I1103 05:10:32.953218 31061 test_util.cc:55] ----------------------------------------------- I1103 05:10:32.953256 31061 test_util.cc:56] Had fatal failures, leaving test files at /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestRunLeaderElection.1415020185628730-31061 [ FAILED ] DistConsensusTest.TestRunLeaderElection (12029 ms) [ RUN ] DistConsensusTest.TestInsertWhenTheQueueIsFull I1103 05:10:32.953908 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master kudu-master --master_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --master_web_port=0 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:32.995378 11830 master_main.cc:27] Initializing master server... I1103 05:10:33.096997 11830 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/master/instance: uuid: "60c564ff49e84d9f952d6a23ce727682" format_stamp: "Formatted at 2014-11-03 13:10:32 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:33.097117 11830 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/master uuid: "60c564ff49e84d9f952d6a23ce727682" format_stamp: "Formatted at 2014-11-03 13:10:32 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:33.100136 11830 master_main.cc:30] Starting master server... I1103 05:10:33.102429 11830 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:33.102463 11830 webserver.cc:117] Document root disabled I1103 05:10:33.102928 11830 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:35013 I1103 05:10:33.103123 11830 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb I1103 05:10:33.109920 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58925 --consensus_entry_cache_size_soft_limit_mb=0 --consensus_entry_cache_size_hard_limit_mb=1 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:33.121655 11857 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: Bootstrap starting. I1103 05:10:33.122068 11857 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one. I1103 05:10:33.122329 11857 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:33.122649 11858 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:33.123019 11857 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: No bootstrap required, opened a new log I1103 05:10:33.129356 11859 tablet_peer.cc:187] Configuration changed for tablet: 11111111111111111111111111111111 in peer: 60c564ff49e84d9f952d6a23ce727682 Changed from: seqno: 0 local: true peers { permanent_uuid: "60c564ff49e84d9f952d6a23ce727682" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "60c564ff49e84d9f952d6a23ce727682" role: LEADER } I1103 05:10:33.129382 11859 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "60c564ff49e84d9f952d6a23ce727682" role: LEADER } I1103 05:10:33.141557 11857 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: Bootstrap starting. I1103 05:10:33.141901 11857 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I1103 05:10:33.142688 11857 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I1103 05:10:33.143221 11857 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:10:33.143424 11865 tablet_peer.cc:187] Configuration changed for tablet: 00000000000000000000000000000000 in peer: 60c564ff49e84d9f952d6a23ce727682 Changed from: seqno: 0 local: true peers { permanent_uuid: "60c564ff49e84d9f952d6a23ce727682" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "60c564ff49e84d9f952d6a23ce727682" role: LEADER } I1103 05:10:33.143455 11865 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "60c564ff49e84d9f952d6a23ce727682" role: LEADER } I1103 05:10:33.143457 11857 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:10:33.143725 11830 master_main.cc:33] Master server successfully started. I1103 05:10:33.148344 11863 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:33.161094 11863 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/instance: uuid: "478297776b8a48caac6c0f5dc1c93d1f" format_stamp: "Formatted at 2014-11-03 13:10:33 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:33.161551 11863 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 uuid: "478297776b8a48caac6c0f5dc1c93d1f" format_stamp: "Formatted at 2014-11-03 13:10:33 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:33.164227 11863 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:33.166996 11863 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:33.167024 11863 webserver.cc:117] Document root disabled I1103 05:10:33.167474 11863 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:35312 I1103 05:10:33.167652 11863 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb I1103 05:10:33.167768 11863 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:33.167870 11863 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:33.169342 11908 heartbeater.cc:227] Registering TS with master... I1103 05:10:33.169363 11908 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:33.169685 11836 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "478297776b8a48caac6c0f5dc1c93d1f" instance_seqno: 1415020233166865 } with Master I1103 05:10:33.171346 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58925 --consensus_entry_cache_size_soft_limit_mb=0 --consensus_entry_cache_size_hard_limit_mb=1 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:33.207795 11912 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:33.229512 11912 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/instance: uuid: "f689340c17cf41b7a94da70779d80089" format_stamp: "Formatted at 2014-11-03 13:10:33 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:33.229630 11912 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 uuid: "f689340c17cf41b7a94da70779d80089" format_stamp: "Formatted at 2014-11-03 13:10:33 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:33.232264 11912 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:33.235013 11912 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:33.235044 11912 webserver.cc:117] Document root disabled I1103 05:10:33.235486 11912 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:42085 I1103 05:10:33.235671 11912 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb I1103 05:10:33.235781 11912 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:33.235884 11912 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:33.237124 11952 heartbeater.cc:227] Registering TS with master... I1103 05:10:33.237144 11952 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:33.237282 11838 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "f689340c17cf41b7a94da70779d80089" instance_seqno: 1415020233234896 } with Master I1103 05:10:33.242198 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58925 --consensus_entry_cache_size_soft_limit_mb=0 --consensus_entry_cache_size_hard_limit_mb=1 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:33.278024 11955 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:33.286959 11955 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/instance: uuid: "30f94f4bda154db2aa3012cf26d4e466" format_stamp: "Formatted at 2014-11-03 13:10:33 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:33.287078 11955 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 uuid: "30f94f4bda154db2aa3012cf26d4e466" format_stamp: "Formatted at 2014-11-03 13:10:33 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:33.289691 11955 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:33.292454 11955 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:33.292484 11955 webserver.cc:117] Document root disabled I1103 05:10:33.292925 11955 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:57466 I1103 05:10:33.293115 11955 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.TestInsertWhenTheQueueIsFull.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb I1103 05:10:33.293226 11955 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:33.293324 11955 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:33.294565 11995 heartbeater.cc:227] Registering TS with master... I1103 05:10:33.294586 11995 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:33.294731 11840 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "30f94f4bda154db2aa3012cf26d4e466" instance_seqno: 1415020233292337 } with Master I1103 05:10:33.303314 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:10:33.303643 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master I1103 05:10:33.305917 11844 catalog_manager.cc:437] CreateTable from {real_user=jenkins, eff_user=} at 127.0.0.1:50435: name: "TestTable" schema { columns { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 3 I1103 05:10:33.308130 11973 tablet_service.cc:264] Processing CreateTablet for tablet d89be74c426142a48e77b1e722f18f99 (table=TestTable [id=a14dbb1cba00475cb469cf60ea9f3945]), range=["", ""] I1103 05:10:33.308326 11930 tablet_service.cc:264] Processing CreateTablet for tablet d89be74c426142a48e77b1e722f18f99 (table=TestTable [id=a14dbb1cba00475cb469cf60ea9f3945]), range=["", ""] I1103 05:10:33.308365 11886 tablet_service.cc:264] Processing CreateTablet for tablet d89be74c426142a48e77b1e722f18f99 (table=TestTable [id=a14dbb1cba00475cb469cf60ea9f3945]), range=["", ""] I1103 05:10:33.390076 11973 ts_tablet_manager.cc:464] Registered tablet d89be74c426142a48e77b1e722f18f99 I1103 05:10:33.390332 12006 ts_tablet_manager.cc:325] Bootstrapping tablet: d89be74c426142a48e77b1e722f18f99 I1103 05:10:33.390450 12006 tablet_bootstrap.cc:234] Tablet d89be74c426142a48e77b1e722f18f99: Bootstrap starting. I1103 05:10:33.390822 12006 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: d89be74c426142a48e77b1e722f18f99 creating new one. I1103 05:10:33.391088 12006 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:33.391264 12007 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:33.391633 12006 tablet_bootstrap.cc:234] Tablet d89be74c426142a48e77b1e722f18f99: No bootstrap required, opened a new log I1103 05:10:33.391681 12006 ts_tablet_manager.cc:330] Time spent tablet d89be74c426142a48e77b1e722f18f99 bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:10:33.392011 12006 raft_consensus.cc:102] T d89be74c426142a48e77b1e722f18f99 P 30f94f4bda154db2aa3012cf26d4e466 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:33.392129 12006 raft_consensus.cc:302] T d89be74c426142a48e77b1e722f18f99 P 30f94f4bda154db2aa3012cf26d4e466 [FOLLOWER]: Becoming Follower/Learner I1103 05:10:33.392186 12006 ts_tablet_manager.cc:350] Time spent starting tablet d89be74c426142a48e77b1e722f18f99: real 0.000s user 0.000s sys 0.000s I1103 05:10:33.396293 11886 ts_tablet_manager.cc:464] Registered tablet d89be74c426142a48e77b1e722f18f99 I1103 05:10:33.396390 12009 ts_tablet_manager.cc:325] Bootstrapping tablet: d89be74c426142a48e77b1e722f18f99 I1103 05:10:33.396507 12009 tablet_bootstrap.cc:234] Tablet d89be74c426142a48e77b1e722f18f99: Bootstrap starting. I1103 05:10:33.396883 12009 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: d89be74c426142a48e77b1e722f18f99 creating new one. I1103 05:10:33.397611 12009 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:33.397773 12010 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:33.398149 12009 tablet_bootstrap.cc:234] Tablet d89be74c426142a48e77b1e722f18f99: No bootstrap required, opened a new log I1103 05:10:33.398197 12009 ts_tablet_manager.cc:330] Time spent tablet d89be74c426142a48e77b1e722f18f99 bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:10:33.398499 12009 raft_consensus.cc:102] T d89be74c426142a48e77b1e722f18f99 P 478297776b8a48caac6c0f5dc1c93d1f [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:33.398612 12009 raft_consensus.cc:302] T d89be74c426142a48e77b1e722f18f99 P 478297776b8a48caac6c0f5dc1c93d1f [FOLLOWER]: Becoming Follower/Learner I1103 05:10:33.398666 12009 ts_tablet_manager.cc:350] Time spent starting tablet d89be74c426142a48e77b1e722f18f99: real 0.000s user 0.000s sys 0.000s I1103 05:10:33.404547 11930 ts_tablet_manager.cc:464] Registered tablet d89be74c426142a48e77b1e722f18f99 I1103 05:10:33.404801 12012 ts_tablet_manager.cc:325] Bootstrapping tablet: d89be74c426142a48e77b1e722f18f99 I1103 05:10:33.404916 12012 tablet_bootstrap.cc:234] Tablet d89be74c426142a48e77b1e722f18f99: Bootstrap starting. I1103 05:10:33.405256 12012 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: d89be74c426142a48e77b1e722f18f99 creating new one. I1103 05:10:33.405503 12012 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:33.405668 12013 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:33.405881 12012 tablet_bootstrap.cc:234] Tablet d89be74c426142a48e77b1e722f18f99: No bootstrap required, opened a new log I1103 05:10:33.405930 12012 ts_tablet_manager.cc:330] Time spent tablet d89be74c426142a48e77b1e722f18f99 bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:10:33.406234 12012 raft_consensus.cc:102] T d89be74c426142a48e77b1e722f18f99 P f689340c17cf41b7a94da70779d80089 [CANDIDATE]: Replica starting. Triggering 0 pending transactions. I1103 05:10:33.406517 12012 raft_consensus.cc:235] T d89be74c426142a48e77b1e722f18f99 P f689340c17cf41b7a94da70779d80089 [LEADER]: Becoming Leader I1103 05:10:33.407827 12012 ts_tablet_manager.cc:350] Time spent starting tablet d89be74c426142a48e77b1e722f18f99: real 0.002s user 0.001s sys 0.000s W1103 05:10:33.907001 12017 raft_consensus_state.cc:463] T d89be74c426142a48e77b1e722f18f99 P f689340c17cf41b7a94da70779d80089 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 W1103 05:10:33.907327 12014 raft_consensus_state.cc:463] T d89be74c426142a48e77b1e722f18f99 P f689340c17cf41b7a94da70779d80089 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:10:33.908675 11983 raft_consensus.cc:608] T d89be74c426142a48e77b1e722f18f99 P 30f94f4bda154db2aa3012cf26d4e466 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:33.908794 11916 consensus_queue.cc:451] Connected to peer: 30f94f4bda154db2aa3012cf26d4e466 whose last received watermark is: term: 0 index: 0 I1103 05:10:33.909131 11896 raft_consensus.cc:608] T d89be74c426142a48e77b1e722f18f99 P 478297776b8a48caac6c0f5dc1c93d1f [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:33.909296 11914 consensus_queue.cc:451] Connected to peer: 478297776b8a48caac6c0f5dc1c93d1f whose last received watermark is: term: 0 index: 0 I1103 05:10:33.909595 11916 raft_consensus_state.cc:456] T d89be74c426142a48e77b1e722f18f99 P f689340c17cf41b7a94da70779d80089 [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I1103 05:10:33.909706 12014 tablet_peer.cc:187] Configuration changed for tablet: d89be74c426142a48e77b1e722f18f99 in peer: f689340c17cf41b7a94da70779d80089 Changed from: seqno: -1 local: false peers { permanent_uuid: "30f94f4bda154db2aa3012cf26d4e466" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43943 } } peers { permanent_uuid: "478297776b8a48caac6c0f5dc1c93d1f" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 50658 } } peers { permanent_uuid: "f689340c17cf41b7a94da70779d80089" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 58179 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "30f94f4bda154db2aa3012cf26d4e466" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43943 } } peers { permanent_uuid: "478297776b8a48caac6c0f5dc1c93d1f" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 50658 } } peers { permanent_uuid: "f689340c17cf41b7a94da70779d80089" role: LEADER last_known_addr { host: "127.0.0.1" port: 58179 } } I1103 05:10:34.237691 11840 catalog_manager.cc:1127] Tablet: d89be74c426142a48e77b1e722f18f99 reported quorum change. New quorum: seqno: 0 local: false peers { permanent_uuid: "30f94f4bda154db2aa3012cf26d4e466" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43943 } } peers { permanent_uuid: "478297776b8a48caac6c0f5dc1c93d1f" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 50658 } } peers { permanent_uuid: "f689340c17cf41b7a94da70779d80089" role: LEADER last_known_addr { host: "127.0.0.1" port: 58179 } } I1103 05:10:34.367271 31061 external_mini_cluster.cc:280] 3 TS(s) registered with Master W1103 05:10:34.367395 11829 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:58925, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:34.368427 31061 raft_consensus-itest.cc:716] Paused one of the replicas, starting to write. I1103 05:10:34.372481 12011 tablet_peer.cc:187] Configuration changed for tablet: d89be74c426142a48e77b1e722f18f99 in peer: 478297776b8a48caac6c0f5dc1c93d1f Changed from: seqno: -1 local: false peers { permanent_uuid: "30f94f4bda154db2aa3012cf26d4e466" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43943 } } peers { permanent_uuid: "478297776b8a48caac6c0f5dc1c93d1f" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 50658 } } peers { permanent_uuid: "f689340c17cf41b7a94da70779d80089" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 58179 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "30f94f4bda154db2aa3012cf26d4e466" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43943 } } peers { permanent_uuid: "478297776b8a48caac6c0f5dc1c93d1f" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 50658 } } peers { permanent_uuid: "f689340c17cf41b7a94da70779d80089" role: LEADER last_known_addr { host: "127.0.0.1" port: 58179 } } W1103 05:10:34.382946 12036 raft_consensus.cc:344] T d89be74c426142a48e77b1e722f18f99 P f689340c17cf41b7a94da70779d80089 [LEADER]: : Could not append replicate request to the queue. Queue is Full. Queue metrics: Consensus queue metrics: Total Ops: 7, All Done Ops: 0, Only Majority Done Ops: 6, In Progress Ops: 56008473, Queue Size (bytes): 921977/1048576 W1103 05:10:34.383469 12036 raft_consensus.cc:344] T d89be74c426142a48e77b1e722f18f99 P f689340c17cf41b7a94da70779d80089 [LEADER]: : Could not append replicate request to the queue. Queue is Full. Queue metrics: Consensus queue metrics: Total Ops: 7, All Done Ops: 0, Only Majority Done Ops: 6, In Progress Ops: 56008473, Queue Size (bytes): 921977/1048576 I1103 05:10:34.384356 12008 tablet_peer.cc:187] Configuration changed for tablet: d89be74c426142a48e77b1e722f18f99 in peer: 30f94f4bda154db2aa3012cf26d4e466 Changed from: seqno: -1 local: false peers { permanent_uuid: "30f94f4bda154db2aa3012cf26d4e466" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43943 } } peers { permanent_uuid: "478297776b8a48caac6c0f5dc1c93d1f" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 50658 } } peers { permanent_uuid: "f689340c17cf41b7a94da70779d80089" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 58179 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "30f94f4bda154db2aa3012cf26d4e466" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43943 } } peers { permanent_uuid: "478297776b8a48caac6c0f5dc1c93d1f" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 50658 } } peers { permanent_uuid: "f689340c17cf41b7a94da70779d80089" role: LEADER last_known_addr { host: "127.0.0.1" port: 58179 } } W1103 05:10:34.485384 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58179, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.530695 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43943, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.612926 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58179, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.650331 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43943, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.717700 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58179, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.751036 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43943, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.827850 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58179, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.860430 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43943, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.946768 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58179, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:34.977879 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43943, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:35.075367 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:58179, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:35.107800 12033 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:43943, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:35.163691 12031 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:50658, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:35.217875 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master with pid 11830 I1103 05:10:35.221271 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 11863 I1103 05:10:35.224803 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 11912 I1103 05:10:35.229044 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 11955 [ OK ] DistConsensusTest.TestInsertWhenTheQueueIsFull (2284 ms) [ RUN ] DistConsensusTest.MultiThreadedInsertWithFailovers I1103 05:10:35.237998 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master kudu-master --master_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/master --master_rpc_bind_addresses=127.0.0.1:0 --master_web_port=0 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:35.288596 12055 master_main.cc:27] Initializing master server... I1103 05:10:35.317351 12055 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/master/instance: uuid: "5c56a2ab267b499eaf2ec364d2e71c1e" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.317466 12055 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/master uuid: "5c56a2ab267b499eaf2ec364d2e71c1e" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.320336 12055 master_main.cc:30] Starting master server... I1103 05:10:35.322454 12055 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:35.322486 12055 webserver.cc:117] Document root disabled I1103 05:10:35.322958 12055 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:49908 I1103 05:10:35.323161 12055 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/master/info.pb I1103 05:10:35.331315 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58236 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:35.346119 12082 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: Bootstrap starting. I1103 05:10:35.346489 12082 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 11111111111111111111111111111111 creating new one. I1103 05:10:35.346753 12082 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:35.347092 12083 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:35.347290 12082 tablet_bootstrap.cc:234] Tablet 11111111111111111111111111111111: No bootstrap required, opened a new log I1103 05:10:35.354570 12084 tablet_peer.cc:187] Configuration changed for tablet: 11111111111111111111111111111111 in peer: 5c56a2ab267b499eaf2ec364d2e71c1e Changed from: seqno: 0 local: true peers { permanent_uuid: "5c56a2ab267b499eaf2ec364d2e71c1e" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "5c56a2ab267b499eaf2ec364d2e71c1e" role: LEADER } I1103 05:10:35.354594 12084 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "5c56a2ab267b499eaf2ec364d2e71c1e" role: LEADER } I1103 05:10:35.377388 12088 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:35.387123 12082 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: Bootstrap starting. I1103 05:10:35.387449 12082 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 00000000000000000000000000000000 creating new one. I1103 05:10:35.387897 12082 tablet_bootstrap.cc:234] Tablet 00000000000000000000000000000000: No bootstrap required, opened a new log I1103 05:10:35.388274 12082 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:10:35.392405 12091 tablet_peer.cc:187] Configuration changed for tablet: 00000000000000000000000000000000 in peer: 5c56a2ab267b499eaf2ec364d2e71c1e Changed from: seqno: 0 local: true peers { permanent_uuid: "5c56a2ab267b499eaf2ec364d2e71c1e" role: LEADER } Changed to: seqno: 1 local: true peers { permanent_uuid: "5c56a2ab267b499eaf2ec364d2e71c1e" role: LEADER } I1103 05:10:35.392447 12091 sys_tables.cc:217] SysTable state changed. New quorum config: seqno: 1 local: true peers { permanent_uuid: "5c56a2ab267b499eaf2ec364d2e71c1e" role: LEADER } I1103 05:10:35.392473 12082 sys_tables.cc:281] SysTable tablet configured and running, proceeding with master startup. I1103 05:10:35.392599 12055 master_main.cc:33] Master server successfully started. I1103 05:10:35.395167 12088 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/instance: uuid: "706cb7ab924d46769952fb8f363e85c5" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.395267 12088 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-0 uuid: "706cb7ab924d46769952fb8f363e85c5" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.397907 12088 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:35.400815 12088 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:35.400857 12088 webserver.cc:117] Document root disabled I1103 05:10:35.401316 12088 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:33559 I1103 05:10:35.401499 12088 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-0/info.pb I1103 05:10:35.401617 12088 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:35.401716 12088 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:35.402916 12133 heartbeater.cc:227] Registering TS with master... I1103 05:10:35.402937 12133 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:35.403260 12061 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" instance_seqno: 1415020235400681 } with Master I1103 05:10:35.405376 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58236 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:35.452332 12137 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:35.463263 12137 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/instance: uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.463381 12137 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-1 uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.466193 12137 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:35.468756 12137 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:35.468799 12137 webserver.cc:117] Document root disabled I1103 05:10:35.469262 12137 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:60978 I1103 05:10:35.469444 12137 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-1/info.pb I1103 05:10:35.469554 12137 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:35.469653 12137 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:35.470903 12177 heartbeater.cc:227] Registering TS with master... I1103 05:10:35.470923 12177 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:35.471060 12063 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" instance_seqno: 1415020235468631 } with Master I1103 05:10:35.475518 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58236 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:35.519400 12180 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:35.528923 12180 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/instance: uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.529949 12180 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-2 uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.532563 12180 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:35.535132 12180 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:35.535161 12180 webserver.cc:117] Document root disabled I1103 05:10:35.535614 12180 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:43082 I1103 05:10:35.535794 12180 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-2/info.pb I1103 05:10:35.535907 12180 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:35.536016 12180 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:35.537267 12220 heartbeater.cc:227] Registering TS with master... I1103 05:10:35.537288 12220 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:35.537436 12065 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" instance_seqno: 1415020235535007 } with Master I1103 05:10:35.543169 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-3 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58236 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-3/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:35.587683 12223 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:35.600612 12223 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-3/instance: uuid: "bca968957e8040d3a9ef18bda42d5a32" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.600733 12223 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-3 uuid: "bca968957e8040d3a9ef18bda42d5a32" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.603448 12223 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:35.606366 12223 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:35.606396 12223 webserver.cc:117] Document root disabled I1103 05:10:35.606861 12223 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:38281 I1103 05:10:35.607045 12223 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-3/info.pb I1103 05:10:35.607167 12223 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:35.607270 12223 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:35.608458 12263 heartbeater.cc:227] Registering TS with master... I1103 05:10:35.608494 12263 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:35.608643 12067 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" instance_seqno: 1415020235606238 } with Master I1103 05:10:35.611292 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-4 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58236 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-4/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:35.656767 12266 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:35.669884 12266 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-4/instance: uuid: "eba314db31e3453c943da62adb9438fa" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.670003 12266 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-4 uuid: "eba314db31e3453c943da62adb9438fa" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.672657 12266 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:35.675451 12266 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:35.675482 12266 webserver.cc:117] Document root disabled I1103 05:10:35.675925 12266 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:49790 I1103 05:10:35.676118 12266 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-4/info.pb I1103 05:10:35.676667 12266 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:35.677016 12266 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:35.678040 12306 heartbeater.cc:227] Registering TS with master... I1103 05:10:35.678076 12306 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:35.678227 12069 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "eba314db31e3453c943da62adb9438fa" instance_seqno: 1415020235675319 } with Master I1103 05:10:35.680120 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-5 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58236 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-5/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:35.724158 12309 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:35.734367 12309 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-5/instance: uuid: "3cf9ab72dd054c579bb9aa6023629daf" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.734483 12309 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-5 uuid: "3cf9ab72dd054c579bb9aa6023629daf" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.737148 12309 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:35.739718 12309 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:35.739747 12309 webserver.cc:117] Document root disabled I1103 05:10:35.740208 12309 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:35443 I1103 05:10:35.740387 12309 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-5/info.pb I1103 05:10:35.740499 12309 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:35.740597 12309 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:35.741819 12349 heartbeater.cc:227] Registering TS with master... I1103 05:10:35.741839 12349 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:35.741968 12061 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" instance_seqno: 1415020235739593 } with Master I1103 05:10:35.747874 31061 external_mini_cluster.cc:359] Running /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server kudu-tablet_server --tablet_server_base_dir=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-6 --tablet_server_rpc_bind_addresses=127.0.0.1:0 --tablet_server_web_port=0 --tablet_server_master_addr=127.0.0.1:58236 --consensus_entry_cache_size_soft_limit_mb=5 --consensus_entry_cache_size_hard_limit_mb=10 --consensus_rpc_timeout_ms=1000 --server_dump_info_path=/tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-6/info.pb --server_dump_info_format=pb --logtostderr --logbuflevel=-1 --webserver_interface=localhost I1103 05:10:35.792116 12352 tablet_server_main.cc:38] Initializing tablet server... I1103 05:10:35.800858 12352 fs_manager.cc:130] Generated new instance metadata in path /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-6/instance: uuid: "e61201183e934af1bfe8aea3112907fe" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.800976 12352 fs_manager.cc:82] Opened local filesystem: /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-6 uuid: "e61201183e934af1bfe8aea3112907fe" format_stamp: "Formatted at 2014-11-03 13:10:35 on boost-static-burst-slave-517.vpc.cloudera.com" I1103 05:10:35.803972 12352 tablet_server_main.cc:41] Starting tablet server... I1103 05:10:35.806891 12352 webserver.cc:108] Starting webserver on localhost:0 I1103 05:10:35.806905 12352 webserver.cc:117] Document root disabled I1103 05:10:35.807348 12352 webserver.cc:197] Webserver started. Bound to: 127.0.0.1:58152 I1103 05:10:35.807513 12352 server_base.cc:166] Dumped server information to /tmp/kudutest-4000/raft_consensus-itest.DistConsensusTest.MultiThreadedInsertWithFailovers.1415020185628730-31061/raft_consensus-itest-cluster/ts-6/info.pb I1103 05:10:35.807611 12352 maintenance_manager.cc:118] MaintenanceManager: targeting memory size of 23.515860 GB I1103 05:10:35.807699 12352 tablet_server_main.cc:44] Tablet server successfully started. I1103 05:10:35.808969 12392 heartbeater.cc:227] Registering TS with master... I1103 05:10:35.808990 12392 heartbeater.cc:233] Sending a full tablet report to master... I1103 05:10:35.809129 12063 ts_manager.cc:66] Registered new tablet server { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" instance_seqno: 1415020235806748 } with Master I1103 05:10:35.816774 31061 external_mini_cluster.cc:280] 7 TS(s) registered with Master I1103 05:10:35.817138 31061 external_mini_cluster.cc:280] 7 TS(s) registered with Master I1103 05:10:35.819711 12067 catalog_manager.cc:437] CreateTable from {real_user=jenkins, eff_user=} at 127.0.0.1:36389: name: "TestTable" schema { columns { name: "key" type: UINT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "int_val" type: UINT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION } } num_replicas: 7 I1103 05:10:35.822106 12198 tablet_service.cc:264] Processing CreateTablet for tablet 17defd19f8b04aeca03e2dd87ac1228b (table=TestTable [id=67d49adb2a1142c298fb4a7523aa47b4]), range=["", ""] I1103 05:10:35.822154 12111 tablet_service.cc:264] Processing CreateTablet for tablet 17defd19f8b04aeca03e2dd87ac1228b (table=TestTable [id=67d49adb2a1142c298fb4a7523aa47b4]), range=["", ""] I1103 05:10:35.822159 12327 tablet_service.cc:264] Processing CreateTablet for tablet 17defd19f8b04aeca03e2dd87ac1228b (table=TestTable [id=67d49adb2a1142c298fb4a7523aa47b4]), range=["", ""] I1103 05:10:35.822556 12155 tablet_service.cc:264] Processing CreateTablet for tablet 17defd19f8b04aeca03e2dd87ac1228b (table=TestTable [id=67d49adb2a1142c298fb4a7523aa47b4]), range=["", ""] I1103 05:10:35.822856 12284 tablet_service.cc:264] Processing CreateTablet for tablet 17defd19f8b04aeca03e2dd87ac1228b (table=TestTable [id=67d49adb2a1142c298fb4a7523aa47b4]), range=["", ""] I1103 05:10:35.822901 12370 tablet_service.cc:264] Processing CreateTablet for tablet 17defd19f8b04aeca03e2dd87ac1228b (table=TestTable [id=67d49adb2a1142c298fb4a7523aa47b4]), range=["", ""] I1103 05:10:35.823293 12241 tablet_service.cc:264] Processing CreateTablet for tablet 17defd19f8b04aeca03e2dd87ac1228b (table=TestTable [id=67d49adb2a1142c298fb4a7523aa47b4]), range=["", ""] I1103 05:10:35.855304 12111 ts_tablet_manager.cc:464] Registered tablet 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.855401 12404 ts_tablet_manager.cc:325] Bootstrapping tablet: 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.855530 12404 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Bootstrap starting. I1103 05:10:35.855954 12404 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 17defd19f8b04aeca03e2dd87ac1228b creating new one. I1103 05:10:35.856215 12404 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:35.856384 12405 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:35.856756 12404 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: No bootstrap required, opened a new log I1103 05:10:35.856806 12404 ts_tablet_manager.cc:330] Time spent tablet 17defd19f8b04aeca03e2dd87ac1228b bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:10:35.857154 12404 raft_consensus.cc:102] T 17defd19f8b04aeca03e2dd87ac1228b P 706cb7ab924d46769952fb8f363e85c5 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:35.857370 12404 raft_consensus.cc:302] T 17defd19f8b04aeca03e2dd87ac1228b P 706cb7ab924d46769952fb8f363e85c5 [FOLLOWER]: Becoming Follower/Learner I1103 05:10:35.857429 12404 ts_tablet_manager.cc:350] Time spent starting tablet 17defd19f8b04aeca03e2dd87ac1228b: real 0.001s user 0.000s sys 0.000s I1103 05:10:35.859953 12370 ts_tablet_manager.cc:464] Registered tablet 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.859956 12198 ts_tablet_manager.cc:464] Registered tablet 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.860003 12284 ts_tablet_manager.cc:464] Registered tablet 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.860054 12407 ts_tablet_manager.cc:325] Bootstrapping tablet: 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.860206 12407 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Bootstrap starting. I1103 05:10:35.860235 12155 ts_tablet_manager.cc:464] Registered tablet 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.860258 12409 ts_tablet_manager.cc:325] Bootstrapping tablet: 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.860337 12408 ts_tablet_manager.cc:325] Bootstrapping tablet: 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.860445 12409 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Bootstrap starting. I1103 05:10:35.860458 12408 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Bootstrap starting. I1103 05:10:35.861320 12410 ts_tablet_manager.cc:325] Bootstrapping tablet: 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.861433 12410 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Bootstrap starting. I1103 05:10:35.862884 12407 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 17defd19f8b04aeca03e2dd87ac1228b creating new one. I1103 05:10:35.862887 12408 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 17defd19f8b04aeca03e2dd87ac1228b creating new one. I1103 05:10:35.862887 12409 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 17defd19f8b04aeca03e2dd87ac1228b creating new one. I1103 05:10:35.862897 12410 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 17defd19f8b04aeca03e2dd87ac1228b creating new one. I1103 05:10:35.863106 12410 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:10:35.863152 12407 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:10:35.863204 12408 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:10:35.863204 12409 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:35.863286 12411 env_posix.cc:511] The filesystem does not support fallocate(). W1103 05:10:35.863404 12414 env_posix.cc:511] The filesystem does not support fallocate(). W1103 05:10:35.863522 12412 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:35.863535 12410 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: No bootstrap required, opened a new log I1103 05:10:35.863574 12410 ts_tablet_manager.cc:330] Time spent tablet 17defd19f8b04aeca03e2dd87ac1228b bootstrap: real 0.002s user 0.000s sys 0.001s I1103 05:10:35.863946 12409 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: No bootstrap required, opened a new log I1103 05:10:35.863950 12410 raft_consensus.cc:102] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:35.863976 12327 ts_tablet_manager.cc:464] Registered tablet 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.864012 12409 ts_tablet_manager.cc:330] Time spent tablet 17defd19f8b04aeca03e2dd87ac1228b bootstrap: real 0.004s user 0.000s sys 0.001s I1103 05:10:35.864068 12407 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: No bootstrap required, opened a new log I1103 05:10:35.864114 12407 ts_tablet_manager.cc:330] Time spent tablet 17defd19f8b04aeca03e2dd87ac1228b bootstrap: real 0.004s user 0.000s sys 0.001s I1103 05:10:35.864168 12410 raft_consensus.cc:302] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [FOLLOWER]: Becoming Follower/Learner I1103 05:10:35.864210 12410 ts_tablet_manager.cc:350] Time spent starting tablet 17defd19f8b04aeca03e2dd87ac1228b: real 0.001s user 0.000s sys 0.000s I1103 05:10:35.864251 12418 ts_tablet_manager.cc:325] Bootstrapping tablet: 17defd19f8b04aeca03e2dd87ac1228b W1103 05:10:35.864322 12413 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:35.864367 12418 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Bootstrap starting. I1103 05:10:35.864380 12409 raft_consensus.cc:102] T 17defd19f8b04aeca03e2dd87ac1228b P eba314db31e3453c943da62adb9438fa [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:35.864509 12407 raft_consensus.cc:102] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:35.864594 12408 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: No bootstrap required, opened a new log I1103 05:10:35.864636 12408 ts_tablet_manager.cc:330] Time spent tablet 17defd19f8b04aeca03e2dd87ac1228b bootstrap: real 0.004s user 0.000s sys 0.001s I1103 05:10:35.864691 12409 raft_consensus.cc:302] T 17defd19f8b04aeca03e2dd87ac1228b P eba314db31e3453c943da62adb9438fa [FOLLOWER]: Becoming Follower/Learner I1103 05:10:35.864739 12418 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 17defd19f8b04aeca03e2dd87ac1228b creating new one. I1103 05:10:35.864765 12409 ts_tablet_manager.cc:350] Time spent starting tablet 17defd19f8b04aeca03e2dd87ac1228b: real 0.001s user 0.000s sys 0.001s I1103 05:10:35.864861 12407 raft_consensus.cc:302] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [FOLLOWER]: Becoming Follower/Learner I1103 05:10:35.864945 12407 ts_tablet_manager.cc:350] Time spent starting tablet 17defd19f8b04aeca03e2dd87ac1228b: real 0.001s user 0.000s sys 0.000s I1103 05:10:35.864984 12408 raft_consensus.cc:102] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:35.865102 12418 log.cc:305] Log is configured to *not* fsync() on all Append() calls I1103 05:10:35.865182 12408 raft_consensus.cc:302] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [FOLLOWER]: Becoming Follower/Learner I1103 05:10:35.865234 12408 ts_tablet_manager.cc:350] Time spent starting tablet 17defd19f8b04aeca03e2dd87ac1228b: real 0.001s user 0.000s sys 0.000s W1103 05:10:35.865277 12420 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:35.865479 12418 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: No bootstrap required, opened a new log I1103 05:10:35.865527 12418 ts_tablet_manager.cc:330] Time spent tablet 17defd19f8b04aeca03e2dd87ac1228b bootstrap: real 0.001s user 0.000s sys 0.001s I1103 05:10:35.865886 12418 raft_consensus.cc:102] T 17defd19f8b04aeca03e2dd87ac1228b P 3cf9ab72dd054c579bb9aa6023629daf [CANDIDATE]: Replica starting. Triggering 0 pending transactions. I1103 05:10:35.866299 12418 raft_consensus.cc:235] T 17defd19f8b04aeca03e2dd87ac1228b P 3cf9ab72dd054c579bb9aa6023629daf [LEADER]: Becoming Leader I1103 05:10:35.868069 12418 ts_tablet_manager.cc:350] Time spent starting tablet 17defd19f8b04aeca03e2dd87ac1228b: real 0.003s user 0.001s sys 0.001s I1103 05:10:35.876529 12241 ts_tablet_manager.cc:464] Registered tablet 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.876621 12430 ts_tablet_manager.cc:325] Bootstrapping tablet: 17defd19f8b04aeca03e2dd87ac1228b I1103 05:10:35.876737 12430 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Bootstrap starting. I1103 05:10:35.877123 12430 tablet_bootstrap.cc:331] No previous blocks or log segments found for tablet: 17defd19f8b04aeca03e2dd87ac1228b creating new one. I1103 05:10:35.877373 12430 log.cc:305] Log is configured to *not* fsync() on all Append() calls W1103 05:10:35.877534 12431 env_posix.cc:511] The filesystem does not support fallocate(). I1103 05:10:35.877735 12430 tablet_bootstrap.cc:234] Tablet 17defd19f8b04aeca03e2dd87ac1228b: No bootstrap required, opened a new log I1103 05:10:35.877784 12430 ts_tablet_manager.cc:330] Time spent tablet 17defd19f8b04aeca03e2dd87ac1228b bootstrap: real 0.001s user 0.001s sys 0.000s I1103 05:10:35.878101 12430 raft_consensus.cc:102] T 17defd19f8b04aeca03e2dd87ac1228b P bca968957e8040d3a9ef18bda42d5a32 [FOLLOWER]: Replica starting. Triggering 0 pending transactions. I1103 05:10:35.878309 12430 raft_consensus.cc:302] T 17defd19f8b04aeca03e2dd87ac1228b P bca968957e8040d3a9ef18bda42d5a32 [FOLLOWER]: Becoming Follower/Learner I1103 05:10:35.878365 12430 ts_tablet_manager.cc:350] Time spent starting tablet 17defd19f8b04aeca03e2dd87ac1228b: real 0.001s user 0.000s sys 0.000s W1103 05:10:36.366631 12423 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 3cf9ab72dd054c579bb9aa6023629daf [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 W1103 05:10:36.367048 12421 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 3cf9ab72dd054c579bb9aa6023629daf [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:10:36.368612 12121 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P 706cb7ab924d46769952fb8f363e85c5 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:36.368881 12313 consensus_queue.cc:451] Connected to peer: 706cb7ab924d46769952fb8f363e85c5 whose last received watermark is: term: 0 index: 0 I1103 05:10:36.369016 12165 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:36.369130 12313 consensus_queue.cc:451] Connected to peer: 7e0acd639fd94f6db3d5ec9112da1d7a whose last received watermark is: term: 0 index: 0 I1103 05:10:36.369407 12380 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. W1103 05:10:36.369467 12313 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 3cf9ab72dd054c579bb9aa6023629daf [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:10:36.369509 12313 consensus_queue.cc:451] Connected to peer: e61201183e934af1bfe8aea3112907fe whose last received watermark is: term: 0 index: 0 I1103 05:10:36.369575 12294 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P eba314db31e3453c943da62adb9438fa [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:36.369741 12311 consensus_queue.cc:451] Connected to peer: eba314db31e3453c943da62adb9438fa whose last received watermark is: term: 0 index: 0 I1103 05:10:36.369849 12208 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. W1103 05:10:36.370107 12313 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 3cf9ab72dd054c579bb9aa6023629daf [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 0 index: 0 New majority replicated is: term: 0 index: 0. Current term is: 1 I1103 05:10:36.370743 12311 raft_consensus_state.cc:456] T 17defd19f8b04aeca03e2dd87ac1228b P 3cf9ab72dd054c579bb9aa6023629daf [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 0 index: 0 New committed index is: term: 1 index: 1 I1103 05:10:36.370848 12421 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 3cf9ab72dd054c579bb9aa6023629daf Changed from: seqno: -1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:36.370971 12312 consensus_queue.cc:451] Connected to peer: 73205f7e568d469a8f11e7c4ab9c4ec2 whose last received watermark is: term: 0 index: 0 I1103 05:10:36.371235 12251 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P bca968957e8040d3a9ef18bda42d5a32 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. I1103 05:10:36.371337 12312 consensus_queue.cc:451] Connected to peer: bca968957e8040d3a9ef18bda42d5a32 whose last received watermark is: term: 0 index: 0 I1103 05:10:36.371582 12416 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 73205f7e568d469a8f11e7c4ab9c4ec2 Changed from: seqno: -1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:36.372130 12432 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: bca968957e8040d3a9ef18bda42d5a32 Changed from: seqno: -1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:36.499392 31061 external_mini_cluster.cc:280] 7 TS(s) registered with Master W1103 05:10:36.499506 12054 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:58236, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:36.500125 31061 raft_consensus-itest.cc:232] Couldn't find the leader and/or replicas. Locations: tablet_id: "17defd19f8b04aeca03e2dd87ac1228b" start_key: "" end_key: "" replicas { ts_info { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" rpc_addresses { host: "127.0.0.1" port: 56156 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" rpc_addresses { host: "127.0.0.1" port: 33750 } } role: FOLLOWER } stale: false I1103 05:10:36.537834 12067 catalog_manager.cc:1127] Tablet: 17defd19f8b04aeca03e2dd87ac1228b reported quorum change. New quorum: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:36.869720 12406 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 706cb7ab924d46769952fb8f363e85c5 Changed from: seqno: -1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:36.869781 12415 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 7e0acd639fd94f6db3d5ec9112da1d7a Changed from: seqno: -1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:36.870569 12417 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: eba314db31e3453c943da62adb9438fa Changed from: seqno: -1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:36.870643 12419 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: e61201183e934af1bfe8aea3112907fe Changed from: seqno: -1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:37.500349 12054 reactor.cc:199] Client call GetTableLocations has no timeout set for connection id: {remote=127.0.0.1:58236, service_name=kudu.master.MasterService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:37.891024 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 12309 W1103 05:10:37.894270 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:37.894362 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:37.894371 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:37.894376 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:37.894398 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:37.894469 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:37.894510 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:37.895079 12463 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:56156, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:37.894402 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:37.895146 12225 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:43480 (request call id 405) could send its response W1103 05:10:37.895258 12270 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:34673 (request call id 570) could send its response W1103 05:10:37.895339 12095 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:49644 (request call id 360) could send its response W1103 05:10:37.895448 12142 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:53431 (request call id 359) could send its response W1103 05:10:37.895536 12356 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:42772 (request call id 327) could send its response W1103 05:10:37.895611 12183 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:49863 (request call id 469) could send its response W1103 05:10:37.896239 12462 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:48875, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:37.896857 12463 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:33750, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:37.897562 12463 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:43873, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:37.898460 12461 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:33158, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:37.899060 12462 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:38575, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:37.899786 12462 reactor.cc:199] Client call RunLeaderElection has no timeout set for connection id: {remote=127.0.0.1:48875, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:37.900151 12209 raft_consensus.cc:180] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [CANDIDATE]: Starting election with quorum: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:37.900329 12209 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Requesting vote from peer eba314db31e3453c943da62adb9438fa I1103 05:10:37.900357 12209 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Requesting vote from peer e61201183e934af1bfe8aea3112907fe I1103 05:10:37.900373 12209 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Requesting vote from peer bca968957e8040d3a9ef18bda42d5a32 I1103 05:10:37.900388 12209 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Requesting vote from peer 706cb7ab924d46769952fb8f363e85c5 I1103 05:10:37.900404 12209 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Requesting vote from peer 3cf9ab72dd054c579bb9aa6023629daf I1103 05:10:37.900418 12209 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Requesting vote from peer 7e0acd639fd94f6db3d5ec9112da1d7a I1103 05:10:37.900688 31061 raft_consensus-itest.cc:817] Killed the first leader. W1103 05:10:37.901231 12603 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.901283 12184 leader_election.cc:218] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: RPC error from VoteRequest() call to peer 3cf9ab72dd054c579bb9aa6023629daf: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) I1103 05:10:37.901767 12251 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica bca968957e8040d3a9ef18bda42d5a32: Leader election vote request: Granting yes vote for candidate 73205f7e568d469a8f11e7c4ab9c4ec2 in term 2. I1103 05:10:37.901782 12384 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica e61201183e934af1bfe8aea3112907fe: Leader election vote request: Granting yes vote for candidate 73205f7e568d469a8f11e7c4ab9c4ec2 in term 2. I1103 05:10:37.901878 12183 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Vote granted by peer bca968957e8040d3a9ef18bda42d5a32 I1103 05:10:37.901927 12184 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Vote granted by peer e61201183e934af1bfe8aea3112907fe I1103 05:10:37.902261 12295 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica eba314db31e3453c943da62adb9438fa: Leader election vote request: Granting yes vote for candidate 73205f7e568d469a8f11e7c4ab9c4ec2 in term 2. I1103 05:10:37.902606 12174 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica 7e0acd639fd94f6db3d5ec9112da1d7a: Leader election vote request: Granting yes vote for candidate 73205f7e568d469a8f11e7c4ab9c4ec2 in term 2. I1103 05:10:37.902724 12182 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Vote granted by peer eba314db31e3453c943da62adb9438fa I1103 05:10:37.902740 12182 leader_election.cc:273] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Election decided. Result: candidate won. I1103 05:10:37.902752 12182 raft_consensus.cc:1029] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [CANDIDATE]: Leader election won for term 2 I1103 05:10:37.902776 12182 raft_consensus.cc:1046] Calling BecomeLeaderUnlocked()... I1103 05:10:37.902783 12182 raft_consensus.cc:235] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [LEADER]: Becoming Leader I1103 05:10:37.902812 12184 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Vote granted by peer 7e0acd639fd94f6db3d5ec9112da1d7a I1103 05:10:37.902896 12122 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica 706cb7ab924d46769952fb8f363e85c5: Leader election vote request: Granting yes vote for candidate 73205f7e568d469a8f11e7c4ab9c4ec2 in term 2. I1103 05:10:37.902997 12184 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2: Term 2 election: Vote granted by peer 706cb7ab924d46769952fb8f363e85c5 W1103 05:10:37.904285 12535 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 1 index: 1007 New majority replicated is: term: 0 index: 0. Current term is: 2 I1103 05:10:37.904310 12296 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P eba314db31e3453c943da62adb9438fa [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 1 index: 1008. Preceding OpId from leader: term: 2 index: 1010. I1103 05:10:37.904356 12252 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P bca968957e8040d3a9ef18bda42d5a32 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 1 index: 1008. Preceding OpId from leader: term: 2 index: 1010. W1103 05:10:37.904371 12602 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.904392 12416 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 1 index: 1007 New majority replicated is: term: 0 index: 0. Current term is: 2 W1103 05:10:37.904412 12184 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:37.904450 12380 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 1 index: 1008. Preceding OpId from leader: term: 2 index: 1010. I1103 05:10:37.904453 12123 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P 706cb7ab924d46769952fb8f363e85c5 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 1 index: 1007. Preceding OpId from leader: term: 2 index: 1010. W1103 05:10:37.904469 12416 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 1 index: 1007 New majority replicated is: term: 0 index: 0. Current term is: 2 I1103 05:10:37.904474 12183 consensus_queue.cc:451] Connected to peer: bca968957e8040d3a9ef18bda42d5a32 whose last received watermark is: term: 1 index: 1008 I1103 05:10:37.904489 12169 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 1 index: 1008. Preceding OpId from leader: term: 2 index: 1010. I1103 05:10:37.904553 12182 consensus_queue.cc:451] Connected to peer: eba314db31e3453c943da62adb9438fa whose last received watermark is: term: 1 index: 1008 I1103 05:10:37.904991 12184 consensus_queue.cc:451] Connected to peer: 7e0acd639fd94f6db3d5ec9112da1d7a whose last received watermark is: term: 1 index: 1008 I1103 05:10:37.905048 12184 consensus_queue.cc:451] Connected to peer: 706cb7ab924d46769952fb8f363e85c5 whose last received watermark is: term: 1 index: 1007 I1103 05:10:37.905128 12184 consensus_queue.cc:451] Connected to peer: e61201183e934af1bfe8aea3112907fe whose last received watermark is: term: 1 index: 1008 W1103 05:10:37.905216 12182 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 1 index: 1007 New majority replicated is: term: 1 index: 1008. Current term is: 2 W1103 05:10:37.905645 12184 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 1 index: 1007 New majority replicated is: term: 1 index: 1008. Current term is: 2 I1103 05:10:37.905897 12183 raft_consensus_state.cc:456] T 17defd19f8b04aeca03e2dd87ac1228b P 73205f7e568d469a8f11e7c4ab9c4ec2 [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 1 index: 1007 New committed index is: term: 2 index: 1011 I1103 05:10:37.906083 12416 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 73205f7e568d469a8f11e7c4ab9c4ec2 Changed from: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:37.907129 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.907172 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.907322 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.907364 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) I1103 05:10:37.910353 12417 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: eba314db31e3453c943da62adb9438fa Changed from: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:37.910658 12419 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: e61201183e934af1bfe8aea3112907fe Changed from: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:37.910675 12432 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: bca968957e8040d3a9ef18bda42d5a32 Changed from: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:37.910722 12406 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 706cb7ab924d46769952fb8f363e85c5 Changed from: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:37.910827 12415 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 7e0acd639fd94f6db3d5ec9112da1d7a Changed from: seqno: 0 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: LEADER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:37.912137 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.912243 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.913144 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.913188 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.913702 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.913748 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.915771 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.915912 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.916612 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.916648 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.916847 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.916888 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.918871 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.918905 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.922082 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.922262 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.922477 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.922528 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.923199 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.923238 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.925454 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.925493 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.928028 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.928113 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.928872 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.928944 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.928949 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.928997 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.935245 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.935287 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.936308 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.936347 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.936435 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.936473 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.936769 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.936815 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.938946 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.938983 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.939986 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.940035 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.940619 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.940654 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.943102 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.943141 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.945078 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.945121 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.945854 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.945901 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.949276 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.949333 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.951083 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.951133 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.951432 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.951462 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.951472 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.951522 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.952383 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.952424 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.956017 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.956060 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.956586 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.956624 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.960809 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.960851 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.962704 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.962749 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.963337 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.963409 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.965103 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.965139 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.965766 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.965801 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.966069 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.966117 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.969156 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.969224 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.970123 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.970160 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.970221 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.970259 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.972810 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.972847 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.975374 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.975416 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.975921 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.975962 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.978730 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.978768 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.980100 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.980101 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.980144 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.980154 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.982041 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.982105 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.985129 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.985174 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.985656 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.985700 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.985988 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.986027 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.987982 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.988157 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.989626 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.989666 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.991873 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.991915 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.993058 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.993108 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.994212 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.994254 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.994765 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.994810 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.996675 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.996723 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.997989 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:37.998028 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.000192 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.000236 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.000772 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.000809 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.003437 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.003510 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.004348 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.004392 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.005180 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.005218 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.008049 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.008102 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.008752 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.008790 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.009834 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.009871 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.012593 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.012629 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.015267 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.015302 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.016417 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.016458 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.017521 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.017563 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.018502 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.018570 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.018935 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.018980 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.021406 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.021443 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.023851 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.023897 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.027158 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.027200 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.027804 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.027894 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.029451 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.029500 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.030119 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.030175 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.031463 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.031510 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.032333 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.032368 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.032543 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.032577 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.034574 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.034615 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.036535 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.036576 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.040720 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.040761 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.044129 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.044176 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.046303 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.046345 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.046555 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.046596 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.047132 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.047173 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.047907 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.047958 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.048403 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.048449 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.049152 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.049188 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.053551 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.053589 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.056138 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.056185 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.058331 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.058365 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.060206 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.060258 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.060705 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.060751 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.060911 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.060976 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.063657 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.063698 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.063956 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.063995 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.064431 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.064479 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.067921 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.067971 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.069711 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.069757 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.072661 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.072720 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.073546 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.073593 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.075508 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.075556 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.077270 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.077311 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.077507 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.077561 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.077765 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.077822 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.081900 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.081944 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.082521 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.082571 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.082603 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.082644 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.084426 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.084463 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.086550 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.086626 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.087465 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.087510 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.088718 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.088764 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.093616 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.093662 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.094215 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.094254 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.094614 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.094663 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.096866 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.096920 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.099252 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.099339 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.099840 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.099889 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.099930 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.099933 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.100867 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.100918 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.103207 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.103255 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.104238 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.104279 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.104564 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.104601 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.107944 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.107980 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.109145 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.109181 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.109815 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.109860 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.112876 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.112923 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.114939 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.114979 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.115923 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.115960 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.118024 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.118067 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.119705 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.119766 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.120828 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.120867 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.121172 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.121211 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.124152 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.124222 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.124771 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.124810 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.127791 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.127835 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.129844 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.129880 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.131414 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.131451 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.131867 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.131911 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.133079 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.133119 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.133121 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.133165 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.135861 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.135902 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.136546 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.136595 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.140249 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.140288 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.147230 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.147271 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.148684 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.148684 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.148736 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.148743 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.149440 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.149476 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.149489 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.149510 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.149531 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.149536 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.149549 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.149570 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.154687 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.154726 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.160035 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.160078 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.161684 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.161741 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.162184 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.162258 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.164819 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.164865 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.165439 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.165477 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.166456 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.166509 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.166628 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.166669 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.166728 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.166772 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.171602 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.171651 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.172502 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.172547 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.173704 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.173823 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.174494 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.174535 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.177837 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.178067 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.178833 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.178874 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.179251 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.179296 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.182397 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.182459 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.183238 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.183282 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.183656 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.183698 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.186753 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.186792 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.188730 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.188812 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.189712 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.189792 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.190122 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.190162 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.192736 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.192780 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.196830 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.196871 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.199257 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.199306 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.199558 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.199601 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.200104 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.200146 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.200889 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.201131 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.201145 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.201169 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.203114 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.203173 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.203647 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.203685 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.206143 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.206183 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.209928 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.209975 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.210734 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.210793 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.211166 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.211204 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.214671 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.214746 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.216837 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.216877 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.217269 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.217314 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.217553 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.217588 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.220371 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.220412 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.220835 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.220882 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.225430 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.225472 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.226018 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.226063 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.229619 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.229663 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.231004 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.231065 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.231600 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.231648 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.233579 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.233640 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.234079 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.234189 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.234192 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.234238 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.237026 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.237062 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.241089 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.241137 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.241886 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.241974 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.242991 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.243026 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.245568 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.245618 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.247455 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.247516 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.248462 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.248507 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.248615 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.248741 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.248749 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.248782 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.252854 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.252910 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.254616 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.254667 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.255107 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.255151 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.257231 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.257280 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.259929 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.259979 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.260723 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.260833 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.260843 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.260877 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.261351 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.261396 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.267153 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.267206 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.268337 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.268404 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.268508 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.268569 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.268626 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.268671 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.270665 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.270710 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.271469 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.271508 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.273473 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.273520 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.275851 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.275895 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.278097 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.278151 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.280033 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.280094 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.282089 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.282150 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.282476 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.282555 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.285823 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.285869 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.286336 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.286387 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.286587 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.286635 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.289129 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.289172 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.290453 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.290495 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.291111 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.291149 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.293416 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.293462 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.294862 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.294899 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.296435 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.296473 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.298959 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.299000 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.299458 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.299495 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.301821 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.301872 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.302593 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.302647 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.305106 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.305169 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.306551 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.306612 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.306905 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.306946 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.307068 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.307104 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.310809 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.310859 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.311928 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.311975 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.313704 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.313760 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.315508 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.315578 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.315706 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.315752 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.318575 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.318635 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.319942 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.319983 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.320332 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.320366 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.322206 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.322247 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.323154 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.323196 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.323400 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.323444 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.327049 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.327110 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.327266 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.327304 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.330785 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.330833 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.331313 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.331393 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.333463 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.333518 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.334175 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.334192 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.334210 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.334231 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.337153 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.337193 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.337575 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.337620 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.339861 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.339901 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.343607 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.343648 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.344420 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.344547 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.345851 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.345901 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.346603 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.346640 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.349925 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.349983 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.351279 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.351363 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.351565 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.351615 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.352012 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.352062 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.353957 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.353994 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.358850 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.358885 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.359220 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.359272 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.359472 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.359516 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.361732 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.361768 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.362512 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.362552 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.363976 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.364033 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.365265 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.365337 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.366932 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.366973 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.373584 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.373631 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.375159 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.375200 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.377033 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.377094 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.377115 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.377193 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.377342 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.377382 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.377877 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.377948 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.378200 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.378237 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.387908 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.387948 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.388659 12507 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.388703 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.388744 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.388793 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.391801 12490 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.391860 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.393038 12524 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.393076 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.393828 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.393874 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.394106 12497 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.394147 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.394448 12502 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.394484 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.399370 12516 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.399407 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.403473 12597 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.403560 12184 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1103 05:10:38.403702 12517 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.403764 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.404789 12520 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.404865 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) I1103 05:10:38.405390 12064 catalog_manager.cc:1127] Tablet: 17defd19f8b04aeca03e2dd87ac1228b reported quorum change. New quorum: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:38.650049 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 12180 W1103 05:10:38.653888 12505 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:38.654109 12514 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:38.654153 12477 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:38.654258 12483 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:38.654363 12508 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:38.654469 12503 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:38.654574 12487 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:38.654674 12480 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:38.655228 12463 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:56156, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:38.655428 12096 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:49679 (request call id 573) could send its response W1103 05:10:38.655530 12356 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:42803 (request call id 603) could send its response W1103 05:10:38.655542 12225 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:43508 (request call id 519) could send its response W1103 05:10:38.655642 12463 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:33750, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:38.655755 12142 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:53469 (request call id 553) could send its response W1103 05:10:38.655884 12463 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:43873, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:38.655905 12270 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:34701 (request call id 728) could send its response W1103 05:10:38.656079 12461 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:33158, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:38.656275 12462 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:38575, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:38.656462 12463 reactor.cc:199] Client call RunLeaderElection has no timeout set for connection id: {remote=127.0.0.1:43873, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:38.656756 12382 raft_consensus.cc:180] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [CANDIDATE]: Starting election with quorum: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:38.656908 12382 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Requesting vote from peer eba314db31e3453c943da62adb9438fa I1103 05:10:38.656934 12382 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Requesting vote from peer bca968957e8040d3a9ef18bda42d5a32 I1103 05:10:38.656950 12382 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Requesting vote from peer 706cb7ab924d46769952fb8f363e85c5 I1103 05:10:38.656975 12382 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Requesting vote from peer 3cf9ab72dd054c579bb9aa6023629daf I1103 05:10:38.656991 12382 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Requesting vote from peer 7e0acd639fd94f6db3d5ec9112da1d7a I1103 05:10:38.657007 12382 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Requesting vote from peer 73205f7e568d469a8f11e7c4ab9c4ec2 W1103 05:10:38.657531 12623 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.657578 12623 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:38.657603 12356 leader_election.cc:218] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: RPC error from VoteRequest() call to peer 3cf9ab72dd054c579bb9aa6023629daf: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:38.657650 12355 leader_election.cc:218] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: RPC error from VoteRequest() call to peer 73205f7e568d469a8f11e7c4ab9c4ec2: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) I1103 05:10:38.657734 31061 raft_consensus-itest.cc:820] Killed the second leader. I1103 05:10:38.658217 12256 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica bca968957e8040d3a9ef18bda42d5a32: Leader election vote request: Granting yes vote for candidate e61201183e934af1bfe8aea3112907fe in term 3. I1103 05:10:38.658236 12128 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica 706cb7ab924d46769952fb8f363e85c5: Leader election vote request: Granting yes vote for candidate e61201183e934af1bfe8aea3112907fe in term 3. I1103 05:10:38.658323 12355 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Vote granted by peer bca968957e8040d3a9ef18bda42d5a32 I1103 05:10:38.658334 12301 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica eba314db31e3453c943da62adb9438fa: Leader election vote request: Granting yes vote for candidate e61201183e934af1bfe8aea3112907fe in term 3. I1103 05:10:38.658359 12356 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Vote granted by peer 706cb7ab924d46769952fb8f363e85c5 I1103 05:10:38.658431 12354 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Vote granted by peer eba314db31e3453c943da62adb9438fa I1103 05:10:38.658450 12354 leader_election.cc:273] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Election decided. Result: candidate won. I1103 05:10:38.658467 12354 raft_consensus.cc:1029] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [CANDIDATE]: Leader election won for term 3 I1103 05:10:38.658501 12354 raft_consensus.cc:1046] Calling BecomeLeaderUnlocked()... I1103 05:10:38.658514 12354 raft_consensus.cc:235] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [LEADER]: Becoming Leader I1103 05:10:38.658908 12167 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica 7e0acd639fd94f6db3d5ec9112da1d7a: Leader election vote request: Granting yes vote for candidate e61201183e934af1bfe8aea3112907fe in term 3. I1103 05:10:38.658993 12356 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe: Term 3 election: Vote granted by peer 7e0acd639fd94f6db3d5ec9112da1d7a W1103 05:10:39.158965 12632 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.159024 12356 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I1103 05:10:39.159032 12125 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P 706cb7ab924d46769952fb8f363e85c5 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 2 index: 2013. Preceding OpId from leader: term: 3 index: 2015. W1103 05:10:39.159137 12632 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:39.159178 12355 consensus_peers.cc:368] Couldn't send request to peer 73205f7e568d469a8f11e7c4ab9c4ec2 for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:39.159237 12629 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 2 index: 2010 New majority replicated is: term: 0 index: 0. Current term is: 3 W1103 05:10:39.159314 12419 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 2 index: 2010 New majority replicated is: term: 0 index: 0. Current term is: 3 I1103 05:10:39.159462 12298 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P eba314db31e3453c943da62adb9438fa [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 2 index: 2014. Preceding OpId from leader: term: 3 index: 2015. I1103 05:10:39.159464 12252 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P bca968957e8040d3a9ef18bda42d5a32 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 2 index: 2011. Preceding OpId from leader: term: 3 index: 2015. I1103 05:10:39.159495 12356 consensus_queue.cc:451] Connected to peer: 706cb7ab924d46769952fb8f363e85c5 whose last received watermark is: term: 2 index: 2013 I1103 05:10:39.159557 12354 consensus_queue.cc:451] Connected to peer: eba314db31e3453c943da62adb9438fa whose last received watermark is: term: 2 index: 2014 I1103 05:10:39.159590 12355 consensus_queue.cc:451] Connected to peer: bca968957e8040d3a9ef18bda42d5a32 whose last received watermark is: term: 2 index: 2011 I1103 05:10:39.159626 12169 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 2 index: 2013. Preceding OpId from leader: term: 3 index: 2015. I1103 05:10:39.159729 12356 consensus_queue.cc:451] Connected to peer: 7e0acd639fd94f6db3d5ec9112da1d7a whose last received watermark is: term: 2 index: 2013 W1103 05:10:39.159981 12354 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 2 index: 2010 New majority replicated is: term: 2 index: 2013. Current term is: 3 W1103 05:10:39.160063 12356 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 2 index: 2010 New majority replicated is: term: 2 index: 2013. Current term is: 3 I1103 05:10:39.160542 12356 raft_consensus_state.cc:456] T 17defd19f8b04aeca03e2dd87ac1228b P e61201183e934af1bfe8aea3112907fe [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 2 index: 2010 New committed index is: term: 3 index: 2015 I1103 05:10:39.160616 12419 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: e61201183e934af1bfe8aea3112907fe Changed from: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:39.656944 12531 lock_manager.cc:345] Waited 1 seconds to obtain row lock on key \x00\x000\xde cur holder: 0x4a18500 W1103 05:10:39.659085 12632 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.659123 12356 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1103 05:10:39.659243 12632 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:39.659278 12355 consensus_peers.cc:368] Couldn't send request to peer 73205f7e568d469a8f11e7c4ab9c4ec2 for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I1103 05:10:39.660725 12406 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 706cb7ab924d46769952fb8f363e85c5 Changed from: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:39.660779 12415 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 7e0acd639fd94f6db3d5ec9112da1d7a Changed from: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:39.660836 12417 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: eba314db31e3453c943da62adb9438fa Changed from: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:39.661206 12432 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: bca968957e8040d3a9ef18bda42d5a32 Changed from: seqno: 1 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: LEADER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:39.664914 12659 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.664963 12660 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.664974 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.665020 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.665894 12661 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.665953 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.666015 12662 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.666067 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.666908 12664 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.666970 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.667024 12663 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.667067 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.667243 12665 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.667296 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.667927 12666 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.667979 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.680451 12659 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:39.680492 12508 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) I1103 05:10:39.680984 12069 catalog_manager.cc:1127] Tablet: 17defd19f8b04aeca03e2dd87ac1228b reported quorum change. New quorum: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:39.687585 12659 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:39.687635 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) I1103 05:10:40.013981 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 12352 W1103 05:10:40.018306 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:40.018337 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:40.018355 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:40.018457 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:40.018471 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:40.018527 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:40.018558 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:40.018645 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: Recv() got EOF from remote (error 108) W1103 05:10:40.019114 12463 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:56156, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:40.019367 12097 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:50010 (request call id 468) could send its response W1103 05:10:40.019386 12227 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:43840 (request call id 591) could send its response W1103 05:10:40.019388 12140 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:53798 (request call id 470) could send its response W1103 05:10:40.019459 12463 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:33750, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:40.019549 12271 connection.cc:299] Connection torn down before Call UpdateConsensus from 127.0.0.1:35033 (request call id 600) could send its response W1103 05:10:40.019664 12461 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:33158, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:40.019863 12462 reactor.cc:199] Client call GetLastOpId has no timeout set for connection id: {remote=127.0.0.1:38575, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:40.020064 12463 reactor.cc:199] Client call RunLeaderElection has no timeout set for connection id: {remote=127.0.0.1:33750, service_name=kudu.consensus.ConsensusService, user_credentials={real_user=jenkins, eff_user=}} I1103 05:10:40.020401 12165 raft_consensus.cc:180] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [CANDIDATE]: Starting election with quorum: seqno: 3 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: CANDIDATE last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:40.020553 12165 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Requesting vote from peer eba314db31e3453c943da62adb9438fa I1103 05:10:40.020577 12165 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Requesting vote from peer e61201183e934af1bfe8aea3112907fe I1103 05:10:40.020593 12165 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Requesting vote from peer bca968957e8040d3a9ef18bda42d5a32 I1103 05:10:40.020614 12165 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Requesting vote from peer 706cb7ab924d46769952fb8f363e85c5 I1103 05:10:40.020630 12165 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Requesting vote from peer 3cf9ab72dd054c579bb9aa6023629daf I1103 05:10:40.020645 12165 leader_election.cc:183] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Requesting vote from peer 73205f7e568d469a8f11e7c4ab9c4ec2 I1103 05:10:40.020769 31061 raft_consensus-itest.cc:823] Killed the third leader. W1103 05:10:40.020864 12668 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.021147 12141 leader_election.cc:218] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: RPC error from VoteRequest() call to peer e61201183e934af1bfe8aea3112907fe: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.021169 12709 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.021211 12141 leader_election.cc:218] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: RPC error from VoteRequest() call to peer 3cf9ab72dd054c579bb9aa6023629daf: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.021211 12709 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.021280 12140 leader_election.cc:218] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: RPC error from VoteRequest() call to peer 73205f7e568d469a8f11e7c4ab9c4ec2: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) I1103 05:10:40.021644 12123 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica 706cb7ab924d46769952fb8f363e85c5: Leader election vote request: Granting yes vote for candidate 7e0acd639fd94f6db3d5ec9112da1d7a in term 4. I1103 05:10:40.021703 12294 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica eba314db31e3453c943da62adb9438fa: Leader election vote request: Granting yes vote for candidate 7e0acd639fd94f6db3d5ec9112da1d7a in term 4. I1103 05:10:40.021729 12141 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Vote granted by peer 706cb7ab924d46769952fb8f363e85c5 I1103 05:10:40.021811 12257 raft_consensus.cc:953] Tablet 17defd19f8b04aeca03e2dd87ac1228b: Replica bca968957e8040d3a9ef18bda42d5a32: Leader election vote request: Granting yes vote for candidate 7e0acd639fd94f6db3d5ec9112da1d7a in term 4. I1103 05:10:40.021812 12139 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Vote granted by peer eba314db31e3453c943da62adb9438fa I1103 05:10:40.021909 12140 leader_election.cc:300] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Vote granted by peer bca968957e8040d3a9ef18bda42d5a32 I1103 05:10:40.021926 12140 leader_election.cc:273] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a: Term 4 election: Election decided. Result: candidate won. I1103 05:10:40.021940 12140 raft_consensus.cc:1029] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [CANDIDATE]: Leader election won for term 4 I1103 05:10:40.021960 12140 raft_consensus.cc:1046] Calling BecomeLeaderUnlocked()... I1103 05:10:40.021968 12140 raft_consensus.cc:235] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [LEADER]: Becoming Leader I1103 05:10:40.522389 12125 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P 706cb7ab924d46769952fb8f363e85c5 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 3 index: 3024. Preceding OpId from leader: term: 4 index: 3027. W1103 05:10:40.522536 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) I1103 05:10:40.522548 12141 consensus_queue.cc:451] Connected to peer: 706cb7ab924d46769952fb8f363e85c5 whose last received watermark is: term: 3 index: 3024 W1103 05:10:40.522619 12141 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:40.522680 12714 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 3 index: 3023 New majority replicated is: term: 0 index: 0. Current term is: 4 W1103 05:10:40.522747 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.522783 12415 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 3 index: 3023 New majority replicated is: term: 0 index: 0. Current term is: 4 W1103 05:10:40.522795 12140 consensus_peers.cc:368] Couldn't send request to peer 73205f7e568d469a8f11e7c4ab9c4ec2 for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:40.523092 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.523136 12141 consensus_peers.cc:368] Couldn't send request to peer e61201183e934af1bfe8aea3112907fe for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1103 05:10:40.523190 12141 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 3 index: 3023 New majority replicated is: term: 0 index: 0. Current term is: 4 I1103 05:10:40.523372 12299 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P eba314db31e3453c943da62adb9438fa [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 3 index: 3026. Preceding OpId from leader: term: 4 index: 3027. I1103 05:10:40.523469 12252 raft_consensus.cc:608] T 17defd19f8b04aeca03e2dd87ac1228b P bca968957e8040d3a9ef18bda42d5a32 [FOLLOWER]: Refusing update from remote peer: Log matching property violated. Last replicated by replica: term: 3 index: 3026. Preceding OpId from leader: term: 4 index: 3027. I1103 05:10:40.523499 12139 consensus_queue.cc:451] Connected to peer: eba314db31e3453c943da62adb9438fa whose last received watermark is: term: 3 index: 3026 I1103 05:10:40.523594 12140 consensus_queue.cc:451] Connected to peer: bca968957e8040d3a9ef18bda42d5a32 whose last received watermark is: term: 3 index: 3026 W1103 05:10:40.523977 12140 raft_consensus_state.cc:463] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [LEADER]: Can't advance the committed index across term boundaries yet. Last committed operation was: term: 3 index: 3023 New majority replicated is: term: 3 index: 3026. Current term is: 4 I1103 05:10:40.524458 12139 raft_consensus_state.cc:456] T 17defd19f8b04aeca03e2dd87ac1228b P 7e0acd639fd94f6db3d5ec9112da1d7a [LEADER]: Advanced the committed_index across terms. Last committed operation was: term: 3 index: 3023 New committed index is: term: 4 index: 3027 I1103 05:10:40.524569 12415 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 7e0acd639fd94f6db3d5ec9112da1d7a Changed from: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 3 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: LEADER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:40.526927 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.526927 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.526996 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.527026 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.529021 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.529081 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.530082 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.530139 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.534390 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.534430 12503 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.534452 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.534492 12505 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.535395 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.535430 12483 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.538539 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.538591 12480 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) I1103 05:10:40.540807 12432 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: bca968957e8040d3a9ef18bda42d5a32 Changed from: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 3 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: LEADER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:40.541291 12406 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: 706cb7ab924d46769952fb8f363e85c5 Changed from: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 3 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: LEADER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } I1103 05:10:40.541375 12417 tablet_peer.cc:187] Configuration changed for tablet: 17defd19f8b04aeca03e2dd87ac1228b in peer: eba314db31e3453c943da62adb9438fa Changed from: seqno: 2 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: LEADER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } Changed to: seqno: 3 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: LEADER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:40.542382 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.542446 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.543000 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.543048 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.543505 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.543541 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.545266 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.545321 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.545444 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.545478 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.545506 12741 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.545544 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.547307 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.547351 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.547503 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.547554 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.551843 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.551890 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.552220 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.552271 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.552947 12741 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.552983 12477 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.553683 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.553704 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.553877 12514 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.553917 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.553972 12508 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.554440 12487 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.557241 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.557272 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.557276 12503 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.557320 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.557356 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.557391 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.559664 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.559726 12505 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.560726 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.560789 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.561614 12741 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.561657 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.563520 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.563577 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.565692 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.565748 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.565779 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.565793 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.565829 12480 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.565845 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.565891 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.565930 12483 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.569311 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.569365 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.569800 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.569855 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.570456 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.570507 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.571533 12741 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.571573 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.573581 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.573642 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.574348 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.574466 12508 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.574470 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.574520 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.575598 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.575644 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.575688 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.575731 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.577857 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.577899 12514 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.578150 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.578199 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.579934 12741 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.579960 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.579993 12487 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.580000 12477 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.581347 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.581387 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.582054 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.582119 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.584030 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.584064 12503 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.584516 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.584604 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.584611 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.584650 12505 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.586573 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.586614 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.587623 12741 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.587687 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.587690 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.587740 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.589738 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.589778 12480 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.591279 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.591287 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.591339 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.591373 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.592455 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.592505 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.594697 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.594741 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.595175 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.595208 12483 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.595402 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.595437 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.595732 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.595784 12508 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.597411 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.597452 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.599138 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.599189 12509 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.599793 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.599812 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.599838 12514 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.599848 12487 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.600478 12741 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.600515 12479 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.604465 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.604508 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.604945 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.604996 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.606606 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.606735 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.606735 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.606786 12513 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.606906 12731 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.606950 12505 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.608001 12741 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.608039 12477 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.608499 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.608551 12486 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.609801 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:40.609838 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts e61201183e934af1bfe8aea3112907fe has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) I1103 05:10:40.611317 12065 catalog_manager.cc:1127] Tablet: 17defd19f8b04aeca03e2dd87ac1228b reported quorum change. New quorum: seqno: 3 local: false peers { permanent_uuid: "706cb7ab924d46769952fb8f363e85c5" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 56156 } } peers { permanent_uuid: "3cf9ab72dd054c579bb9aa6023629daf" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 46986 } } peers { permanent_uuid: "73205f7e568d469a8f11e7c4ab9c4ec2" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 48875 } } peers { permanent_uuid: "7e0acd639fd94f6db3d5ec9112da1d7a" role: LEADER last_known_addr { host: "127.0.0.1" port: 33750 } } peers { permanent_uuid: "e61201183e934af1bfe8aea3112907fe" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 43873 } } peers { permanent_uuid: "eba314db31e3453c943da62adb9438fa" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 33158 } } peers { permanent_uuid: "bca968957e8040d3a9ef18bda42d5a32" role: FOLLOWER last_known_addr { host: "127.0.0.1" port: 38575 } } W1103 05:10:40.613168 12730 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.613203 12503 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.613342 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.613384 12485 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.614574 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.614634 12518 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.615865 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.615988 12511 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.617043 12732 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.617140 12480 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.619078 12742 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.619139 12514 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.619797 12729 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.619832 12483 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.619910 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.619997 12493 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 3cf9ab72dd054c579bb9aa6023629daf has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:40.624699 12744 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.624760 12508 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.626425 12743 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:40.626473 12487 meta_cache.cc:159] Replica 17defd19f8b04aeca03e2dd87ac1228b on ts 73205f7e568d469a8f11e7c4ab9c4ec2 has failed: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:41.020907 31061 thread.cc:320] Waited for 1000ms trying to join with ts-test0 (tid 12471) W1103 05:10:41.022651 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:41.022694 12141 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1103 05:10:41.022862 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:41.022914 12140 consensus_peers.cc:368] Couldn't send request to peer 73205f7e568d469a8f11e7c4ab9c4ec2 for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1103 05:10:41.023026 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:41.023059 12141 consensus_peers.cc:368] Couldn't send request to peer e61201183e934af1bfe8aea3112907fe for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1103 05:10:41.030503 12415 log.cc:507] Time spent Append to log took a long time: real 0.167s user 0.000s sys 0.000s W1103 05:10:41.032014 12417 log.cc:507] Time spent Append to log took a long time: real 0.168s user 0.000s sys 0.000s W1103 05:10:41.032230 12406 log.cc:507] Time spent Append to log took a long time: real 0.168s user 0.000s sys 0.000s W1103 05:10:41.032835 12432 log.cc:507] Time spent Append to log took a long time: real 0.168s user 0.000s sys 0.000s W1103 05:10:41.105762 12463 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:33750, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:41.429349 12463 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:56156, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:41.522815 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:41.522877 12141 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1103 05:10:41.523005 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:41.523043 12140 consensus_peers.cc:368] Couldn't send request to peer 73205f7e568d469a8f11e7c4ab9c4ec2 for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1103 05:10:41.523169 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:41.523212 12141 consensus_peers.cc:368] Couldn't send request to peer e61201183e934af1bfe8aea3112907fe for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1103 05:10:41.843008 12463 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:33750, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:42.022977 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:42.023035 12141 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1103 05:10:42.023125 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:42.023162 12140 consensus_peers.cc:368] Couldn't send request to peer 73205f7e568d469a8f11e7c4ab9c4ec2 for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1103 05:10:42.023300 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:42.023344 12141 consensus_peers.cc:368] Couldn't send request to peer e61201183e934af1bfe8aea3112907fe for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1103 05:10:42.146703 12463 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:56156, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:42.523144 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:42.523212 12141 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1103 05:10:42.523277 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:42.523315 12140 consensus_peers.cc:368] Couldn't send request to peer 73205f7e568d469a8f11e7c4ab9c4ec2 for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1103 05:10:42.523433 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:42.523466 12141 consensus_peers.cc:368] Couldn't send request to peer e61201183e934af1bfe8aea3112907fe for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W1103 05:10:42.524427 12461 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:33158, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:42.903982 12462 reactor.cc:199] Client call Scan has no timeout set for connection id: {remote=127.0.0.1:38575, service_name=kudu.tserver.TabletServerService, user_credentials={real_user=jenkins, eff_user=}} W1103 05:10:43.023221 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111) W1103 05:10:43.023270 12141 consensus_peers.cc:368] Couldn't send request to peer 3cf9ab72dd054c579bb9aa6023629daf for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:46986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1103 05:10:43.023356 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111) W1103 05:10:43.023391 12140 consensus_peers.cc:368] Couldn't send request to peer 73205f7e568d469a8f11e7c4ab9c4ec2 for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:48875: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W1103 05:10:43.023571 12720 negotiation.cc:195] Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111) W1103 05:10:43.023604 12141 consensus_peers.cc:368] Couldn't send request to peer e61201183e934af1bfe8aea3112907fe for tablet 17defd19f8b04aeca03e2dd87ac1228b ops [] Status: Network error: client connection negotiation failed: client Connection to 127.0.0.1:43873: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. I1103 05:10:43.275537 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-master with pid 12055 I1103 05:10:43.278439 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 12088 I1103 05:10:43.283679 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 12137 I1103 05:10:43.288715 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 12223 I1103 05:10:43.292896 31061 external_mini_cluster.cc:413] Killing /data1/jenkins-workspace/kudu-flaky-tests/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/build/release/kudu-tablet_server with pid 12266 [ OK ] DistConsensusTest.MultiThreadedInsertWithFailovers (8069 ms) [----------] 8 tests from DistConsensusTest (57677 ms total) [----------] Global test environment tear-down [==========] 8 tests from 1 test case ran. (57677 ms total) [ PASSED ] 7 tests. [ FAILED ] 1 test, listed below: [ FAILED ] DistConsensusTest.TestRunLeaderElection 1 FAILED TEST