I0208 05:35:22.085009 2721 ts_tablet_manager.cc:628] T 5384471d823e46929029f9ff6ce212a3 P c713ac498df040caa897d3229214baa3: Initiating tablet copy from peer 804ce32e20854d08bc8afb076a200caa (127.1.33.132:59055) I0208 05:35:22.086117 1906 ts_tablet_manager.cc:1282] T 75551612df7a40e99ac411e870833b82 P 53455a5280d24e8891948dbcc85dfc18: tablet deleted: last-logged OpId: (unknown) I0208 05:35:22.086164 1906 log.cc:981] T 75551612df7a40e99ac411e870833b82 P 53455a5280d24e8891948dbcc85dfc18: Deleting WAL directory at /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-5/wal/wals/75551612df7a40e99ac411e870833b82 I0208 05:35:22.086612 1178 catalog_manager.cc:2828] TS 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134:50851): tablet 75551612df7a40e99ac411e870833b82 (table RemoveReplaceInCycle [id=9f3ed66e3cd04d15a800fba5caeea97b]) successfully deleted I0208 05:35:22.087009 1560 raft_consensus.cc:2655] T 75551612df7a40e99ac411e870833b82 P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, NON_VOTER 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } } } I0208 05:35:22.087779 1555 raft_consensus.cc:2655] T 7a5ba8948e964f079c1ceb6b105d9749 P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, NON_VOTER 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } } peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } } I0208 05:35:22.088560 1740 consensus_queue.cc:531] T 75551612df7a40e99ac411e870833b82 P 6fde43dee5034c2bb430e75a84246e24 [LEADER]: The tablet replica hosted on peer c713ac498df040caa897d3229214baa3 has failed W0208 05:35:22.089059 1740 consensus_peers.cc:434] T 75551612df7a40e99ac411e870833b82 P 6fde43dee5034c2bb430e75a84246e24 -> Peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): Couldn't send request to peer c713ac498df040caa897d3229214baa3 for tablet 75551612df7a40e99ac411e870833b82. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/wal/wals/75551612df7a40e99ac411e870833b82/.kudutmp.newsegmentyKvxuF (532549632 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 1 times. I0208 05:35:22.089493 1558 raft_consensus.cc:1060] T 75551612df7a40e99ac411e870833b82 P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Refusing update from remote peer 6fde43dee5034c2bb430e75a84246e24: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch) I0208 05:35:22.089808 1192 catalog_manager.cc:3523] T 7a5ba8948e964f079c1ceb6b105d9749 P e9617fe0e9c74b5d84c285682d08a4f3 reported cstate change: config changed from index 20 to 21, NON_VOTER 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129) evicted. New cstate: current_term: 1 leader_uuid: "6fde43dee5034c2bb430e75a84246e24" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } } peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } } I0208 05:35:22.090771 2672 ts_tablet_manager.cc:628] T 2c74a90e180847ddbdcaed674f61f91a P 53455a5280d24e8891948dbcc85dfc18: Initiating tablet copy from peer 6fde43dee5034c2bb430e75a84246e24 (127.1.33.133:60304) I0208 05:35:22.090822 2672 tablet_copy_client.cc:200] T 2c74a90e180847ddbdcaed674f61f91a P 53455a5280d24e8891948dbcc85dfc18: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0208 05:35:22.090888 2672 tablet_copy_client.cc:237] T 2c74a90e180847ddbdcaed674f61f91a P 53455a5280d24e8891948dbcc85dfc18: tablet copy: Beginning tablet copy session from remote peer at address 127.1.33.133:60304 I0208 05:35:22.091368 1825 tablet_copy_service.cc:133] P 6fde43dee5034c2bb430e75a84246e24: Received BeginTabletCopySession request for tablet 2c74a90e180847ddbdcaed674f61f91a from peer 53455a5280d24e8891948dbcc85dfc18 ({username='slave'} at 127.1.33.134:40855) I0208 05:35:22.091413 1192 catalog_manager.cc:2850] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 7a5ba8948e964f079c1ceb6b105d9749 on 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507) (TS 94b9b1edd1bc46de8c96338ebd2bae71 not found in new config with opid_index 21) I0208 05:35:22.091434 1825 tablet_copy_service.cc:154] P 6fde43dee5034c2bb430e75a84246e24: Beginning new tablet copy session on tablet 2c74a90e180847ddbdcaed674f61f91a from peer 53455a5280d24e8891948dbcc85dfc18 at {username='slave'} at 127.1.33.134:40855: session id = 53455a5280d24e8891948dbcc85dfc18-2c74a90e180847ddbdcaed674f61f91a I0208 05:35:22.092048 1294 tablet_service.cc:797] Processing DeleteTablet for tablet 7a5ba8948e964f079c1ceb6b105d9749 with delete_type TABLET_DATA_TOMBSTONED (TS 94b9b1edd1bc46de8c96338ebd2bae71 not found in new config with opid_index 21) from {username='slave'} at 127.0.0.1:48634 I0208 05:35:22.092162 1825 tablet_copy_source_session.cc:202] T 2c74a90e180847ddbdcaed674f61f91a P 6fde43dee5034c2bb430e75a84246e24: Tablet Copy: opened 0 blocks and 1 log segments I0208 05:35:22.092196 1294 ts_tablet_manager.cc:1269] T 7a5ba8948e964f079c1ceb6b105d9749 P 94b9b1edd1bc46de8c96338ebd2bae71: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0208 05:35:22.092885 2672 ts_tablet_manager.cc:1269] T 2c74a90e180847ddbdcaed674f61f91a P 53455a5280d24e8891948dbcc85dfc18: Deleting tablet data with delete state TABLET_DATA_COPYING I0208 05:35:22.093125 2721 tablet_copy_client.cc:200] T 5384471d823e46929029f9ff6ce212a3 P c713ac498df040caa897d3229214baa3: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0208 05:35:22.093248 2721 tablet_copy_client.cc:237] T 5384471d823e46929029f9ff6ce212a3 P c713ac498df040caa897d3229214baa3: tablet copy: Beginning tablet copy session from remote peer at address 127.1.33.132:59055 I0208 05:35:22.093921 1702 tablet_copy_service.cc:133] P 804ce32e20854d08bc8afb076a200caa: Received BeginTabletCopySession request for tablet 5384471d823e46929029f9ff6ce212a3 from peer c713ac498df040caa897d3229214baa3 ({username='slave'} at 127.1.33.130:35414) I0208 05:35:22.094002 1702 tablet_copy_service.cc:154] P 804ce32e20854d08bc8afb076a200caa: Beginning new tablet copy session on tablet 5384471d823e46929029f9ff6ce212a3 from peer c713ac498df040caa897d3229214baa3 at {username='slave'} at 127.1.33.130:35414: session id = c713ac498df040caa897d3229214baa3-5384471d823e46929029f9ff6ce212a3 I0208 05:35:22.094518 2672 ts_tablet_manager.cc:1282] T 2c74a90e180847ddbdcaed674f61f91a P 53455a5280d24e8891948dbcc85dfc18: tablet deleted: last-logged OpId: (unknown) W0208 05:35:22.094635 2672 data_dirs.cc:268] Insufficient disk space under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-5/data/data: creation of new data blocks under this path can be retried after 30 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-5/data/data (527040512 bytes available vs 527083724 bytes reserved) (error 28) I0208 05:35:22.094818 1702 tablet_copy_source_session.cc:202] T 5384471d823e46929029f9ff6ce212a3 P 804ce32e20854d08bc8afb076a200caa: Tablet Copy: opened 0 blocks and 1 log segments I0208 05:35:22.094992 1192 catalog_manager.cc:3162] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 7a5ba8948e964f079c1ceb6b105d9749 (attempt 1) W0208 05:35:22.095295 1738 consensus_peers.cc:419] T 2c74a90e180847ddbdcaed674f61f91a P 6fde43dee5034c2bb430e75a84246e24 -> Peer 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134:50851): Unable to begin Tablet Copy on peer: error { code: UNKNOWN_ERROR status { code: IO_ERROR message: "Could not create a new directory group for tablet copy: All healthy data directories are full" posix_code: 28 } } I0208 05:35:22.096118 2093 consensus_queue.cc:1008] T 75551612df7a40e99ac411e870833b82 P 6fde43dee5034c2bb430e75a84246e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 }, Status: LMP_MISMATCH, Last received: 1.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s I0208 05:35:22.096225 1803 consensus_peers.cc:456] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 -> Peer e9617fe0e9c74b5d84c285682d08a4f3 (127.1.33.131:42173): Closing peer: e9617fe0e9c74b5d84c285682d08a4f3 I0208 05:35:22.096230 1294 ts_tablet_manager.cc:1282] T 7a5ba8948e964f079c1ceb6b105d9749 P 94b9b1edd1bc46de8c96338ebd2bae71: tablet deleted: last-logged OpId: (unknown) I0208 05:35:22.096254 1803 consensus_peers.cc:456] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 -> Peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): Closing peer: c713ac498df040caa897d3229214baa3 I0208 05:35:22.096267 1294 log.cc:981] T 7a5ba8948e964f079c1ceb6b105d9749 P 94b9b1edd1bc46de8c96338ebd2bae71: Deleting WAL directory at /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-0/wal/wals/7a5ba8948e964f079c1ceb6b105d9749 I0208 05:35:22.096379 1803 consensus_queue.cc:196] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } } peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: NON_VOTER last_known_addr { host: "127.1.33.134" port: 50851 } attrs { promote: true } } I0208 05:35:22.096846 1180 catalog_manager.cc:2828] TS 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): tablet 7a5ba8948e964f079c1ceb6b105d9749 (table RemoveReplaceInCycle [id=9f3ed66e3cd04d15a800fba5caeea97b]) successfully deleted W0208 05:35:22.097787 2715 log.cc:1128] Error closing log: Illegal state: Log not open. State: 0 I0208 05:35:22.098070 2721 ts_tablet_manager.cc:1269] T 5384471d823e46929029f9ff6ce212a3 P c713ac498df040caa897d3229214baa3: Deleting tablet data with delete state TABLET_DATA_COPYING E0208 05:35:22.098086 2715 ts_tablet_manager.cc:972] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P c713ac498df040caa897d3229214baa3: Tablet failed to bootstrap: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/wal/wals/3e7e1cb378fc4e2e8ea807f5f7c1474b/.kudutmp.newsegmentkDbS2m (527056896 bytes available vs 527083724 bytes reserved) (error 28) I0208 05:35:22.098110 2715 ts_tablet_manager.cc:951] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P c713ac498df040caa897d3229214baa3: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.000s I0208 05:35:22.098132 2715 tablet_replica.cc:261] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P c713ac498df040caa897d3229214baa3: stopping tablet replica I0208 05:35:22.098150 2715 raft_consensus.cc:2014] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P c713ac498df040caa897d3229214baa3 [term 1 LEARNER]: Raft consensus shutting down. I0208 05:35:22.098189 2715 raft_consensus.cc:2041] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P c713ac498df040caa897d3229214baa3 [term 1 LEARNER]: Raft consensus is shut down! W0208 05:35:22.099516 1616 consensus_peers.cc:434] T 5384471d823e46929029f9ff6ce212a3 P 804ce32e20854d08bc8afb076a200caa -> Peer 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134:50851): Couldn't send request to peer 53455a5280d24e8891948dbcc85dfc18 for tablet 5384471d823e46929029f9ff6ce212a3. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-5/wal/wals/5384471d823e46929029f9ff6ce212a3/.kudutmp.newsegmentKQpNey (530976768 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 3 times. I0208 05:35:22.100136 1702 tablet_copy_service.cc:323] P 804ce32e20854d08bc8afb076a200caa: Request end of tablet copy session c713ac498df040caa897d3229214baa3-3e7e1cb378fc4e2e8ea807f5f7c1474b received from {username='slave'} at 127.1.33.130:35414 I0208 05:35:22.100183 1702 tablet_copy_service.cc:411] P 804ce32e20854d08bc8afb076a200caa: Ending tablet copy session c713ac498df040caa897d3229214baa3-3e7e1cb378fc4e2e8ea807f5f7c1474b on tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b with peer c713ac498df040caa897d3229214baa3 W0208 05:35:22.101374 1738 consensus_peers.cc:434] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 -> Peer 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134:50851): Couldn't send request to peer 53455a5280d24e8891948dbcc85dfc18 for tablet 7a5ba8948e964f079c1ceb6b105d9749. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. Retrying in the next heartbeat period. Already tried 1 times. I0208 05:35:22.102541 2721 ts_tablet_manager.cc:1282] T 5384471d823e46929029f9ff6ce212a3 P c713ac498df040caa897d3229214baa3: tablet deleted: last-logged OpId: (unknown) W0208 05:35:22.102707 2721 data_dirs.cc:268] Insufficient disk space under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/data/data: creation of new data blocks under this path can be retried after 30 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/data/data (527048704 bytes available vs 527083724 bytes reserved) (error 28) I0208 05:35:22.103135 2093 raft_consensus.cc:2655] T 75551612df7a40e99ac411e870833b82 P 6fde43dee5034c2bb430e75a84246e24 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, NON_VOTER 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: NON_VOTER last_known_addr { host: "127.1.33.134" port: 50851 } attrs { promote: true } } } W0208 05:35:22.103615 1740 consensus_peers.cc:434] T 2c74a90e180847ddbdcaed674f61f91a P 6fde43dee5034c2bb430e75a84246e24 -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Couldn't send request to peer 94b9b1edd1bc46de8c96338ebd2bae71 for tablet 2c74a90e180847ddbdcaed674f61f91a. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-0/wal/wals/2c74a90e180847ddbdcaed674f61f91a/.kudutmp.newsegment4bcW4e (532787200 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 2 times. W0208 05:35:22.103173 1618 consensus_peers.cc:419] T 5384471d823e46929029f9ff6ce212a3 P 804ce32e20854d08bc8afb076a200caa -> Peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): Unable to begin Tablet Copy on peer: error { code: UNKNOWN_ERROR status { code: IO_ERROR message: "Could not create a new directory group for tablet copy: All healthy data directories are full" posix_code: 28 } } I0208 05:35:22.105113 1179 catalog_manager.cc:3077] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 75551612df7a40e99ac411e870833b82 with cas_config_opid_index 23: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) W0208 05:35:22.105643 1618 consensus_peers.cc:434] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Couldn't send request to peer 94b9b1edd1bc46de8c96338ebd2bae71 for tablet c114703435b2445db87e6fe66954b244. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-0/wal/wals/c114703435b2445db87e6fe66954b244/.kudutmp.newsegmentgh8wIH (532660224 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 4 times. I0208 05:35:22.106410 1560 raft_consensus.cc:1060] T 7a5ba8948e964f079c1ceb6b105d9749 P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Refusing update from remote peer 6fde43dee5034c2bb430e75a84246e24: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch) I0208 05:35:22.106961 1192 catalog_manager.cc:3523] T 75551612df7a40e99ac411e870833b82 P 6fde43dee5034c2bb430e75a84246e24 reported cstate change: config changed from index 23 to 24, NON_VOTER 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134) added. New cstate: current_term: 1 leader_uuid: "6fde43dee5034c2bb430e75a84246e24" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } health_report { overall_health: FAILED } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: NON_VOTER last_known_addr { host: "127.1.33.134" port: 50851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0208 05:35:22.107838 1560 raft_consensus.cc:2655] T 75551612df7a40e99ac411e870833b82 P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, NON_VOTER 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: NON_VOTER last_known_addr { host: "127.1.33.134" port: 50851 } attrs { promote: true } } } W0208 05:35:22.108280 1374 consensus_peers.cc:434] T 1b7f8f654d84446a82896dff694974d3 P c713ac498df040caa897d3229214baa3 -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Couldn't send request to peer 94b9b1edd1bc46de8c96338ebd2bae71 for tablet 1b7f8f654d84446a82896dff694974d3. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-0/wal/wals/1b7f8f654d84446a82896dff694974d3/.kudutmp.newsegmentYfoemY (532758528 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 2 times. I0208 05:35:22.110942 1740 consensus_queue.cc:531] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 [LEADER]: The tablet replica hosted on peer c713ac498df040caa897d3229214baa3 has failed I0208 05:35:22.111143 2093 consensus_queue.cc:1008] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 }, Status: LMP_MISMATCH, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s W0208 05:35:22.111413 1740 consensus_peers.cc:434] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 -> Peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): Couldn't send request to peer c713ac498df040caa897d3229214baa3 for tablet 7a5ba8948e964f079c1ceb6b105d9749. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/wal/wals/7a5ba8948e964f079c1ceb6b105d9749/.kudutmp.newsegmentvjsBul (532217856 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 1 times. I0208 05:35:22.113528 2378 raft_consensus.cc:2655] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, NON_VOTER 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } } peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: NON_VOTER last_known_addr { host: "127.1.33.134" port: 50851 } attrs { promote: true } } } I0208 05:35:22.115650 1179 catalog_manager.cc:3077] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7a5ba8948e964f079c1ceb6b105d9749 with cas_config_opid_index 21: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0208 05:35:22.117171 1560 raft_consensus.cc:2655] T 7a5ba8948e964f079c1ceb6b105d9749 P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, NON_VOTER 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } } peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: NON_VOTER last_known_addr { host: "127.1.33.134" port: 50851 } attrs { promote: true } } } I0208 05:35:22.117200 1192 catalog_manager.cc:3523] T 7a5ba8948e964f079c1ceb6b105d9749 P 6fde43dee5034c2bb430e75a84246e24 reported cstate change: config changed from index 21 to 22, NON_VOTER 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134) added. New cstate: current_term: 1 leader_uuid: "6fde43dee5034c2bb430e75a84246e24" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: VOTER last_known_addr { host: "127.1.33.133" port: 60304 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c713ac498df040caa897d3229214baa3" member_type: VOTER last_known_addr { host: "127.1.33.130" port: 48208 } health_report { overall_health: FAILED } } peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: NON_VOTER last_known_addr { host: "127.1.33.134" port: 50851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0208 05:35:22.121757 1618 consensus_queue.cc:531] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa [LEADER]: The tablet replica hosted on peer c713ac498df040caa897d3229214baa3 has failed W0208 05:35:22.122017 1618 consensus_peers.cc:434] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): Couldn't send request to peer c713ac498df040caa897d3229214baa3 for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/wal/wals/3e7e1cb378fc4e2e8ea807f5f7c1474b/.kudutmp.newsegmentkDbS2m (527056896 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 2 times. I0208 05:35:22.122364 2695 ts_tablet_manager.cc:628] T 1b7f8f654d84446a82896dff694974d3 P 804ce32e20854d08bc8afb076a200caa: Initiating tablet copy from peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208) I0208 05:35:22.122411 2695 tablet_copy_client.cc:200] T 1b7f8f654d84446a82896dff694974d3 P 804ce32e20854d08bc8afb076a200caa: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0208 05:35:22.122469 2695 tablet_copy_client.cc:237] T 1b7f8f654d84446a82896dff694974d3 P 804ce32e20854d08bc8afb076a200caa: tablet copy: Beginning tablet copy session from remote peer at address 127.1.33.130:48208 I0208 05:35:22.122958 1452 tablet_copy_service.cc:133] P c713ac498df040caa897d3229214baa3: Received BeginTabletCopySession request for tablet 1b7f8f654d84446a82896dff694974d3 from peer 804ce32e20854d08bc8afb076a200caa ({username='slave'} at 127.1.33.132:38019) I0208 05:35:22.123000 2672 ts_tablet_manager.cc:628] T 907689aece8743f0a1d844e1a433dcb6 P 53455a5280d24e8891948dbcc85dfc18: Initiating tablet copy from peer 6fde43dee5034c2bb430e75a84246e24 (127.1.33.133:60304) I0208 05:35:22.123030 1452 tablet_copy_service.cc:154] P c713ac498df040caa897d3229214baa3: Beginning new tablet copy session on tablet 1b7f8f654d84446a82896dff694974d3 from peer 804ce32e20854d08bc8afb076a200caa at {username='slave'} at 127.1.33.132:38019: session id = 804ce32e20854d08bc8afb076a200caa-1b7f8f654d84446a82896dff694974d3 I0208 05:35:22.123046 2672 tablet_copy_client.cc:200] T 907689aece8743f0a1d844e1a433dcb6 P 53455a5280d24e8891948dbcc85dfc18: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0208 05:35:22.123101 2672 tablet_copy_client.cc:237] T 907689aece8743f0a1d844e1a433dcb6 P 53455a5280d24e8891948dbcc85dfc18: tablet copy: Beginning tablet copy session from remote peer at address 127.1.33.133:60304 I0208 05:35:22.123574 1825 tablet_copy_service.cc:133] P 6fde43dee5034c2bb430e75a84246e24: Received BeginTabletCopySession request for tablet 907689aece8743f0a1d844e1a433dcb6 from peer 53455a5280d24e8891948dbcc85dfc18 ({username='slave'} at 127.1.33.134:40855) I0208 05:35:22.123641 1825 tablet_copy_service.cc:154] P 6fde43dee5034c2bb430e75a84246e24: Beginning new tablet copy session on tablet 907689aece8743f0a1d844e1a433dcb6 from peer 53455a5280d24e8891948dbcc85dfc18 at {username='slave'} at 127.1.33.134:40855: session id = 53455a5280d24e8891948dbcc85dfc18-907689aece8743f0a1d844e1a433dcb6 I0208 05:35:22.123693 1452 tablet_copy_source_session.cc:202] T 1b7f8f654d84446a82896dff694974d3 P c713ac498df040caa897d3229214baa3: Tablet Copy: opened 0 blocks and 1 log segments I0208 05:35:22.124202 1825 tablet_copy_source_session.cc:202] T 907689aece8743f0a1d844e1a433dcb6 P 6fde43dee5034c2bb430e75a84246e24: Tablet Copy: opened 0 blocks and 1 log segments I0208 05:35:22.124835 2672 ts_tablet_manager.cc:1269] T 907689aece8743f0a1d844e1a433dcb6 P 53455a5280d24e8891948dbcc85dfc18: Deleting tablet data with delete state TABLET_DATA_COPYING I0208 05:35:22.124864 2695 ts_tablet_manager.cc:1269] T 1b7f8f654d84446a82896dff694974d3 P 804ce32e20854d08bc8afb076a200caa: Deleting tablet data with delete state TABLET_DATA_COPYING I0208 05:35:22.125751 1193 catalog_manager.cc:3274] Sending ChangeConfig:REMOVE_PEER on tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b (attempt 1) I0208 05:35:22.126611 2672 ts_tablet_manager.cc:1282] T 907689aece8743f0a1d844e1a433dcb6 P 53455a5280d24e8891948dbcc85dfc18: tablet deleted: last-logged OpId: (unknown) W0208 05:35:22.126706 2672 data_dirs.cc:268] Insufficient disk space under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-5/data/data: creation of new data blocks under this path can be retried after 30 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-5/data/data (527036416 bytes available vs 527083724 bytes reserved) (error 28) I0208 05:35:22.126755 2695 ts_tablet_manager.cc:1282] T 1b7f8f654d84446a82896dff694974d3 P 804ce32e20854d08bc8afb076a200caa: tablet deleted: last-logged OpId: (unknown) W0208 05:35:22.126802 1618 consensus_peers.cc:434] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Couldn't send request to peer 94b9b1edd1bc46de8c96338ebd2bae71 for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-0/wal/wals/3e7e1cb378fc4e2e8ea807f5f7c1474b/.kudutmp.newsegmentrTRuxN (530558976 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 3 times. W0208 05:35:22.126864 2695 data_dirs.cc:268] Insufficient disk space under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-3/data/data: creation of new data blocks under this path can be retried after 30 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-3/data/data (527036416 bytes available vs 527083724 bytes reserved) (error 28) W0208 05:35:22.127576 1374 consensus_peers.cc:419] T 1b7f8f654d84446a82896dff694974d3 P c713ac498df040caa897d3229214baa3 -> Peer 804ce32e20854d08bc8afb076a200caa (127.1.33.132:59055): Unable to begin Tablet Copy on peer: error { code: UNKNOWN_ERROR status { code: IO_ERROR message: "Could not create a new directory group for tablet copy: All healthy data directories are full" posix_code: 28 } } W0208 05:35:22.127982 1738 consensus_peers.cc:419] T 907689aece8743f0a1d844e1a433dcb6 P 6fde43dee5034c2bb430e75a84246e24 -> Peer 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134:50851): Unable to begin Tablet Copy on peer: error { code: UNKNOWN_ERROR status { code: IO_ERROR message: "Could not create a new directory group for tablet copy: All healthy data directories are full" posix_code: 28 } } I0208 05:35:22.128715 1682 consensus_peers.cc:456] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): Closing peer: c713ac498df040caa897d3229214baa3 I0208 05:35:22.128746 1682 consensus_peers.cc:456] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Closing peer: 94b9b1edd1bc46de8c96338ebd2bae71 I0208 05:35:22.128756 1682 consensus_peers.cc:456] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer e9617fe0e9c74b5d84c285682d08a4f3 (127.1.33.131:42173): Closing peer: e9617fe0e9c74b5d84c285682d08a4f3 I0208 05:35:22.128881 1682 consensus_queue.cc:196] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } I0208 05:35:22.130092 1560 raft_consensus.cc:1060] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Refusing update from remote peer 804ce32e20854d08bc8afb076a200caa: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch) I0208 05:35:22.130491 2686 consensus_queue.cc:1008] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 }, Status: LMP_MISMATCH, Last received: 1.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s I0208 05:35:22.131405 1618 consensus_queue.cc:531] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa [LEADER]: The tablet replica hosted on peer 94b9b1edd1bc46de8c96338ebd2bae71 has failed W0208 05:35:22.131616 1618 consensus_peers.cc:434] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Couldn't send request to peer 94b9b1edd1bc46de8c96338ebd2bae71 for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-0/wal/wals/3e7e1cb378fc4e2e8ea807f5f7c1474b/.kudutmp.newsegmentrTRuxN (530558976 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 1 times. I0208 05:35:22.131749 1618 consensus_queue.cc:531] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa [LEADER]: The tablet replica hosted on peer c713ac498df040caa897d3229214baa3 has failed W0208 05:35:22.131786 1618 consensus_peers.cc:434] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa -> Peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): Couldn't send request to peer c713ac498df040caa897d3229214baa3 for tablet c114703435b2445db87e6fe66954b244. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/wal/wals/c114703435b2445db87e6fe66954b244/.kudutmp.newsegmentEJ6uop (527101952 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 3 times. I0208 05:35:22.132438 2686 raft_consensus.cc:2655] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, NON_VOTER c713ac498df040caa897d3229214baa3 (127.1.33.130) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } } I0208 05:35:22.133116 1193 catalog_manager.cc:3274] Sending ChangeConfig:REMOVE_PEER on tablet c114703435b2445db87e6fe66954b244 (attempt 1) I0208 05:35:22.134042 1180 catalog_manager.cc:3077] ChangeConfig:REMOVE_PEER RPC for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0208 05:35:22.134104 1682 consensus_peers.cc:456] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa -> Peer c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): Closing peer: c713ac498df040caa897d3229214baa3 I0208 05:35:22.134130 1682 consensus_peers.cc:456] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Closing peer: 94b9b1edd1bc46de8c96338ebd2bae71 I0208 05:35:22.134138 1682 consensus_peers.cc:456] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa -> Peer 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134:50851): Closing peer: 53455a5280d24e8891948dbcc85dfc18 I0208 05:35:22.134249 1682 consensus_queue.cc:196] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: VOTER last_known_addr { host: "127.1.33.134" port: 50851 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } I0208 05:35:22.134243 1560 raft_consensus.cc:2655] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, NON_VOTER c713ac498df040caa897d3229214baa3 (127.1.33.130) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } } I0208 05:35:22.134860 1193 catalog_manager.cc:3523] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa reported cstate change: config changed from index 16 to 17, NON_VOTER c713ac498df040caa897d3229214baa3 (127.1.33.130) evicted. New cstate: current_term: 1 leader_uuid: "804ce32e20854d08bc8afb076a200caa" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } health_report { overall_health: FAILED } } peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } health_report { overall_health: HEALTHY } } } I0208 05:35:22.135891 1925 raft_consensus.cc:1060] T c114703435b2445db87e6fe66954b244 P 53455a5280d24e8891948dbcc85dfc18 [term 1 FOLLOWER]: Refusing update from remote peer 804ce32e20854d08bc8afb076a200caa: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch) I0208 05:35:22.136214 1618 consensus_queue.cc:531] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa [LEADER]: The tablet replica hosted on peer 94b9b1edd1bc46de8c96338ebd2bae71 has failed I0208 05:35:22.136546 2247 consensus_queue.cc:1008] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa [LEADER]: Connected to new peer: Peer: permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: VOTER last_known_addr { host: "127.1.33.134" port: 50851 }, Status: LMP_MISMATCH, Last received: 1.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s I0208 05:35:22.136943 1193 catalog_manager.cc:2850] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b on c713ac498df040caa897d3229214baa3 (127.1.33.130:48208) (TS c713ac498df040caa897d3229214baa3 not found in new config with opid_index 17) W0208 05:35:22.137275 1618 consensus_peers.cc:434] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Couldn't send request to peer 94b9b1edd1bc46de8c96338ebd2bae71 for tablet c114703435b2445db87e6fe66954b244. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-0/wal/wals/c114703435b2445db87e6fe66954b244/.kudutmp.newsegmentgh8wIH (532660224 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 1 times. I0208 05:35:22.137370 1417 tablet_service.cc:797] Processing DeleteTablet for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b with delete_type TABLET_DATA_TOMBSTONED (TS c713ac498df040caa897d3229214baa3 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:42348 I0208 05:35:22.137482 1417 ts_tablet_manager.cc:1269] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P c713ac498df040caa897d3229214baa3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0208 05:35:22.139135 1417 ts_tablet_manager.cc:1282] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P c713ac498df040caa897d3229214baa3: tablet deleted: last-logged OpId: (unknown) I0208 05:35:22.139168 1417 log.cc:981] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P c713ac498df040caa897d3229214baa3: Deleting WAL directory at /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/wal/wals/3e7e1cb378fc4e2e8ea807f5f7c1474b I0208 05:35:22.139806 1180 catalog_manager.cc:2828] TS c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b (table RemoveReplaceInCycle [id=9f3ed66e3cd04d15a800fba5caeea97b]) successfully deleted I0208 05:35:22.140499 2075 raft_consensus.cc:2655] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER c713ac498df040caa897d3229214baa3 (127.1.33.130) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: VOTER last_known_addr { host: "127.1.33.134" port: 50851 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } } I0208 05:35:22.141070 1925 raft_consensus.cc:2655] T c114703435b2445db87e6fe66954b244 P 53455a5280d24e8891948dbcc85dfc18 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER c713ac498df040caa897d3229214baa3 (127.1.33.130) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: VOTER last_known_addr { host: "127.1.33.134" port: 50851 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } } I0208 05:35:22.141793 1180 catalog_manager.cc:3077] ChangeConfig:REMOVE_PEER RPC for tablet c114703435b2445db87e6fe66954b244 with cas_config_opid_index 18: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0208 05:35:22.142971 1192 catalog_manager.cc:3523] T c114703435b2445db87e6fe66954b244 P 53455a5280d24e8891948dbcc85dfc18 reported cstate change: config changed from index 18 to 19, NON_VOTER c713ac498df040caa897d3229214baa3 (127.1.33.130) evicted. New cstate: current_term: 1 leader_uuid: "804ce32e20854d08bc8afb076a200caa" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: VOTER last_known_addr { host: "127.1.33.134" port: 50851 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } } I0208 05:35:22.144737 1192 catalog_manager.cc:2850] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet c114703435b2445db87e6fe66954b244 on c713ac498df040caa897d3229214baa3 (127.1.33.130:48208) (TS c713ac498df040caa897d3229214baa3 not found in new config with opid_index 19) I0208 05:35:22.145347 1417 tablet_service.cc:797] Processing DeleteTablet for tablet c114703435b2445db87e6fe66954b244 with delete_type TABLET_DATA_TOMBSTONED (TS c713ac498df040caa897d3229214baa3 not found in new config with opid_index 19) from {username='slave'} at 127.0.0.1:42348 I0208 05:35:22.145452 1417 ts_tablet_manager.cc:1269] T c114703435b2445db87e6fe66954b244 P c713ac498df040caa897d3229214baa3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0208 05:35:22.146980 1417 ts_tablet_manager.cc:1282] T c114703435b2445db87e6fe66954b244 P c713ac498df040caa897d3229214baa3: tablet deleted: last-logged OpId: (unknown) I0208 05:35:22.147012 1417 log.cc:981] T c114703435b2445db87e6fe66954b244 P c713ac498df040caa897d3229214baa3: Deleting WAL directory at /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-1/wal/wals/c114703435b2445db87e6fe66954b244 I0208 05:35:22.147502 1180 catalog_manager.cc:2828] TS c713ac498df040caa897d3229214baa3 (127.1.33.130:48208): tablet c114703435b2445db87e6fe66954b244 (table RemoveReplaceInCycle [id=9f3ed66e3cd04d15a800fba5caeea97b]) successfully deleted F0208 05:35:22.152496 2721 ts_tablet_manager.cc:563] T 5384471d823e46929029f9ff6ce212a3 P c713ac498df040caa897d3229214baa3: Tablet Copy: Found tablet in TABLET_DATA_COPYING state during StartTabletCopy() *** Check failure stack trace: *** @ 0x7f22d7fa92fd google::LogMessage::Fail() at ??:0 @ 0x7f22d7fab1bd google::LogMessage::SendToLog() at ??:0 @ 0x7f22d7fa8e39 google::LogMessage::Flush() at ??:0 @ 0x7f22d7fabc5f google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f22e1fa6ed7 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0 I0208 05:35:22.157768 1193 catalog_manager.cc:3162] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b (attempt 1) @ 0x7f22e1fb296f kudu::tserver::TabletCopyRunnable::Run() at ??:0 I0208 05:35:22.159916 1193 catalog_manager.cc:3162] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b (attempt 1) I0208 05:35:22.160234 1193 catalog_manager.cc:3162] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c114703435b2445db87e6fe66954b244 (attempt 1) @ 0x7f22d9c9a1f5 kudu::ThreadPool::DispatchThread() at ??:0 W0208 05:35:22.160392 1740 consensus_peers.cc:434] T 907689aece8743f0a1d844e1a433dcb6 P 6fde43dee5034c2bb430e75a84246e24 -> Peer 804ce32e20854d08bc8afb076a200caa (127.1.33.132:59055): Couldn't send request to peer 804ce32e20854d08bc8afb076a200caa for tablet 907689aece8743f0a1d844e1a433dcb6. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-3/wal/wals/907689aece8743f0a1d844e1a433dcb6/.kudutmp.newsegmentyaWWwx (533024768 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 2 times. I0208 05:35:22.161020 1681 consensus_peers.cc:456] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Closing peer: 94b9b1edd1bc46de8c96338ebd2bae71 I0208 05:35:22.161052 1681 consensus_peers.cc:456] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer e9617fe0e9c74b5d84c285682d08a4f3 (127.1.33.131:42173): Closing peer: e9617fe0e9c74b5d84c285682d08a4f3 @ 0x7f22d9ca9069 boost::_mfi::mf0<>::operator()() at ??:0 I0208 05:35:22.161149 1681 consensus_queue.cc:196] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: NON_VOTER last_known_addr { host: "127.1.33.133" port: 60304 } attrs { promote: true } } @ 0x7f22d9ca8fd0 boost::_bi::list1<>::operator()<>() at ??:0 I0208 05:35:22.163374 1618 consensus_queue.cc:531] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa [LEADER]: The tablet replica hosted on peer 94b9b1edd1bc46de8c96338ebd2bae71 has failed W0208 05:35:22.163671 1618 consensus_peers.cc:434] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Couldn't send request to peer 94b9b1edd1bc46de8c96338ebd2bae71 for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 8388608 bytes under path /tmp/run_tha_testVVnrkg/test-tmp/raft_consensus_nonvoter-itest.RaftConsensusNonVoterITest.RemoveReplaceInCycle.1518068116111680-1158/raft_consensus-itest-cluster/ts-0/wal/wals/3e7e1cb378fc4e2e8ea807f5f7c1474b/.kudutmp.newsegmentrTRuxN (530558976 bytes available vs 527083724 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 1 times. @ 0x7f22d9ca8f7a boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f22d9ca8d5d boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 I0208 05:35:22.165627 2665 ts_tablet_manager.cc:628] T 53e35500d5c64334bda8438fed0fcd66 P e9617fe0e9c74b5d84c285682d08a4f3: Initiating tablet copy from peer 804ce32e20854d08bc8afb076a200caa (127.1.33.132:59055) I0208 05:35:22.165681 2665 tablet_copy_client.cc:200] T 53e35500d5c64334bda8438fed0fcd66 P e9617fe0e9c74b5d84c285682d08a4f3: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0208 05:35:22.165742 2665 tablet_copy_client.cc:237] T 53e35500d5c64334bda8438fed0fcd66 P e9617fe0e9c74b5d84c285682d08a4f3: tablet copy: Beginning tablet copy session from remote peer at address 127.1.33.132:59055 I0208 05:35:22.166235 1702 tablet_copy_service.cc:133] P 804ce32e20854d08bc8afb076a200caa: Received BeginTabletCopySession request for tablet 53e35500d5c64334bda8438fed0fcd66 from peer e9617fe0e9c74b5d84c285682d08a4f3 ({username='slave'} at 127.1.33.131:39238) I0208 05:35:22.166291 1702 tablet_copy_service.cc:154] P 804ce32e20854d08bc8afb076a200caa: Beginning new tablet copy session on tablet 53e35500d5c64334bda8438fed0fcd66 from peer e9617fe0e9c74b5d84c285682d08a4f3 at {username='slave'} at 127.1.33.131:39238: session id = e9617fe0e9c74b5d84c285682d08a4f3-53e35500d5c64334bda8438fed0fcd66 @ 0x7f22db98d018 boost::function0<>::operator()() at ??:0 I0208 05:35:22.166965 1180 catalog_manager.cc:3094] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b with cas_config_opid_index 17: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 804ce32e20854d08bc8afb076a200caa (127.1.33.132:59055) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } }. Pending config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: VOTER last_known_addr { host: "127.1.33.131" port: 42173 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } peers { permanent_uuid: "6fde43dee5034c2bb430e75a84246e24" member_type: NON_VOTER last_known_addr { host: "127.1.33.133" port: 60304 } attrs { promote: true } } I0208 05:35:22.167019 1180 catalog_manager.cc:2628] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e7e1cb378fc4e2e8ea807f5f7c1474b with cas_config_opid_index 17 with a delay of 32 ms (attempt = 1) I0208 05:35:22.167307 1681 consensus_peers.cc:456] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa -> Peer 94b9b1edd1bc46de8c96338ebd2bae71 (127.1.33.129:46507): Closing peer: 94b9b1edd1bc46de8c96338ebd2bae71 I0208 05:35:22.167332 1681 consensus_peers.cc:456] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa -> Peer 53455a5280d24e8891948dbcc85dfc18 (127.1.33.134:50851): Closing peer: 53455a5280d24e8891948dbcc85dfc18 I0208 05:35:22.167419 1681 consensus_queue.cc:196] T c114703435b2445db87e6fe66954b244 P 804ce32e20854d08bc8afb076a200caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "804ce32e20854d08bc8afb076a200caa" member_type: VOTER last_known_addr { host: "127.1.33.132" port: 59055 } } peers { permanent_uuid: "53455a5280d24e8891948dbcc85dfc18" member_type: VOTER last_known_addr { host: "127.1.33.134" port: 50851 } } peers { permanent_uuid: "94b9b1edd1bc46de8c96338ebd2bae71" member_type: VOTER last_known_addr { host: "127.1.33.129" port: 46507 } } peers { permanent_uuid: "e9617fe0e9c74b5d84c285682d08a4f3" member_type: NON_VOTER last_known_addr { host: "127.1.33.131" port: 42173 } attrs { promote: true } } @ 0x7f22d9c8bdfd kudu::Thread::SuperviseThread() at ??:0 I0208 05:35:22.168368 1560 raft_consensus.cc:1060] T 3e7e1cb378fc4e2e8ea807f5f7c1474b P e9617fe0e9c74b5d84c285682d08a4f3 [term 1 FOLLOWER]: Refusing update from remote peer 804ce32e20854d08bc8afb076a200caa: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch) I0208 05:35:22.168565 1702 tablet_copy_source_session.cc:202] T 53e35500d5c64334bda8438fed0fcd66 P 804ce32e20854d08bc8afb076a200caa: Tablet Copy: opened 0 blocks and 1 log segments @ 0x7f22ddc68184 start_thread at ??:0